Launch script multislice

[blahBlahhhJ]

Things I'm going to add to launch.py

1. Migrate some commands API from tpu-vm to queued-resources to work better with multislice training
2. Delete the instance if command fail (regardless of autodelete) so that it won't eat up all the quota after many retries

[blahBlahhhJ]

@dlwh @rjpower FYI

[rjpower]

For (1) SGTM, let me know if I can help. Something simple that just runs queued-resources create and then polls until the instance is started seems like a good first step.
For (2) don't we normally want to reuse the instance? That's much faster for me in practice than delete/restart. (Or is this for batch job, in which case, sure, makes sense as an option).

[blahBlahhhJ]

In my experience when an instance is preempted (suspended), then you cannot create another one with the same name. The suspended instance is going to stay there and take up the quota.

[dlwh]

I think you all are taking about slightly different things. If the node is still up we want to reuse. Otherwise we have to delete the node and queued resource and recreate (Russell this is one of the main reasons i do not like queued resources…)

…

On Wed, Jun 12, 2024 at 4:50 PM Jason Wang ***@***.***> wrote: In my experience when an instance is preempted (suspended), then you cannot create another one with the same name. The suspended instance is going to stay there and take up the quota. — Reply to this email directly, view it on GitHub <#629 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAACLIK7VD5ZXBJVFF6F5JDZHDNCXAVCNFSM6AAAAABJHJHEZ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRUGA4DQMZTHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[blahBlahhhJ]

@rjpower is there a way to specify which branch to run?

[rjpower]

Not specifically. The script packages whatever is in your current directory, so you can just checkout the branch you want to run.

…

On Thu, Jun 13, 2024, 2:33 PM Jason Wang ***@***.***> wrote: @rjpower <https://github.com/rjpower> is there a way to specify which branch to run? — Reply to this email directly, view it on GitHub <#629 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEUHZ57IXFWILWAU4CD6WLZHIF33AVCNFSM6AAAAABJHJHEZ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRWHAZTCNZUG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>