Keep artifacts for canceled jobs

[MichelML]

I am aware this is probably not the place to open this issue, but it seems like artifacts are not kept when a job is canceled. But why? I just finished a 2 days running job and wanted to have access to my best model, but the artifacts are not showing up potentially because I canceled the job. If this is the case, this is very frustrating.

[BartoszCki]

Hi @MichelML

I'm sorry to hear that. I'm not sure if we can restore your artifacts. I've sent a memo to people who can answer that question, though.

[MichelML]

Thanks @BartoszCki , and also the higher level point I'm trying to make is the artifacts SHOULD remain alive even if someone is cancelling a job

[BartoszCki]

@MichelML
Could you try again, please? If it does still does not work, could you share the job's ID?

PS: IDK what caused lack of access to your artifacts but by default they should be kept after job is canceled or after any error.

[MichelML]

Thanks for your time @BartoszCki . Sorry for the frustration of yesterday - usually not a good time to log an issue. There is a chance that the mistake is on my side. I stopped my job around the time at which the old best model and new best model were swapping - which means the old file is erased and a new file is created. It may be that I stopped the job exactly after the old file was deleted and before the new one got created.

I will run some additional tests to confirm if the artifacts are kept or not when the job is canceled and will get back to you if needed. We can close this issue for now.

Sorry again.

[BartoszCki]

Hey, @MichelML no problem :)

If you have any problems/questions in the future, please don't hesitate to post it here or contact tech support.

Cheers!