Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

image classification needs cancel #4632

Closed
LittleLittleCloud opened this issue Jan 6, 2020 · 7 comments · Fixed by #4650
Closed

image classification needs cancel #4632

LittleLittleCloud opened this issue Jan 6, 2020 · 7 comments · Fixed by #4650
Assignees
Labels

Comments

@LittleLittleCloud
Copy link
Contributor

@LittleLittleCloud LittleLittleCloud commented Jan 6, 2020

Hi, is there any plan to add cancel support for image classification API so that we can cancel an image classification training by passing in a cancellation token?

related issue

@antoniovs1029 antoniovs1029 self-assigned this Jan 6, 2020
@antoniovs1029

This comment has been minimized.

Copy link
Member

@antoniovs1029 antoniovs1029 commented Jan 7, 2020

Hi @LittleLittleCloud , so it's still not clear to me what is being asked, and so if it would be a quick fix or a major feature request.

Can you please share code (and perhaps a dataset) where you "cancel and switch to another scenario" (as mentioned in the issue you've linked to)? I would like to know how you're currently cancelling the training.

Particularly, it's strange that you mention that there's a lot of memory being used and that's not released after cancelling. Perhaps you are pointing to a memory leak?

Thanks.

@LittleLittleCloud

This comment has been minimized.

Copy link
Contributor Author

@LittleLittleCloud LittleLittleCloud commented Jan 7, 2020

It's not a memory leak, the thing is our program allows user to cancel training after it start, But we can't stop an image training after it starts because current Image Classification API doesn't support cancellation token.

The way we use Image Classification is through AutoML, there's a trainer in MulticlassificationExperiment , and that's how it be called

@codemzs

This comment has been minimized.

Copy link
Member

@codemzs codemzs commented Jan 7, 2020

This should be an easy fix. Please refer to the below PRs where we have added cancellation mechanism in other trainers. You can invoke cancellation via mlcontext cancel function that is exposed in experimental nuget. First, someone needs go add cancellation checkpoints, test, verify with model builder, commit, ship.

https://github.com/dotnet/machinelearning/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aclosed+cancellation++

@antoniovs1029

This comment has been minimized.

Copy link
Member

@antoniovs1029 antoniovs1029 commented Jan 7, 2020

Would doing what @codemzs has just said (to expose cancelation via mlContext.CancelExecution()) be enough for you, @LittleLittleCloud ?

@LittleLittleCloud

This comment has been minimized.

Copy link
Contributor Author

@LittleLittleCloud LittleLittleCloud commented Jan 8, 2020

If we can cancel using CancelExecution() that also works. ( As long as there's a way to cancel training)

@codemzs

This comment has been minimized.

Copy link
Member

@codemzs codemzs commented Jan 8, 2020

Yep, we will need to add a check point in the training loop, it’s just few lines of change plus testing.

@antoniovs1029

This comment has been minimized.

Copy link
Member

@antoniovs1029 antoniovs1029 commented Jan 8, 2020

Ok, so I will work on this. Thanks for your suggestions, @codemzs !

@harishsk harishsk added the P1 label Jan 9, 2020
@ganik ganik added P2 and removed P1 labels Jan 10, 2020
antoniovs1029 added a commit that referenced this issue Jan 17, 2020
* Added CheckAlive() Checkpoints in ImageClassificationTrainer
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
5 participants
You can’t perform that action at this time.