Improve fine-tuning checkpoint support

[artek0chumak]

Issue #ENG-18828 , #ENG-18829

Describe your changes

Improve fine-tuning checkpoint support with two features:

• Add new command to show all checkpoints for the jobs
• Add new argument to start a job from the checkpoint

[mryab]

Suggested change

	click.echo("\nTo download a checkpoint, use cmd: together fine-tuning download")
	click.echo("\nTo download a checkpoint, use `together fine-tuning download`")

[mryab]

I'd suggest to format the timestamps to strings only in this function. As you can see in your own example, truncating the seconds etc. can lead to a result where the order of checkpoints becomes incorrect: therefore, it's better to sort by the integer timestamp value. Furthermore, users of the Python CLI might want to apply other datetime formats to display the values in their code, so our current behavior is too destructive

[mryab]

What should be the behavior if the user passes a FT job ID without a colon? I believe this expression won't match, perhaps it's best to rename it to clarify that it only handles job IDs with step numbers?

[mryab]

step=0 will put the code in a surprise :)

Suggested change

	checkpoint_name = f"{id}:{step}" if step else id
	checkpoint_name = f"{id}:{step}" if step is not None else id

[mryab]

For correct sorting, timestamps should not be strings at this point

[mryab]

What do you think about listing the step here as well, like "Intermediate (step 20)"? It's not completely obvious how to interpret the table now, but if we add a bit of extra information, it will be clearer to the user what the names stand for

[mryab]

Suggested change

	from_checkpoint (str, optional): The checkpoint to be used in the fine-tuning.
	from_checkpoint (str, optional): The checkpoint identifier to continue training from a previous fine-tuning job.

[mryab]

Suggested change

	from_checkpoint (str, optional): The checkpoint to be used in the fine-tuning.
	from_checkpoint (str, optional): The checkpoint identifier to continue training from a previous fine-tuning job.