Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Training: cache feature vectors for 2nd epoch onward #80

Merged
merged 4 commits into from
Mar 26, 2024

Conversation

StephenChan
Copy link
Member

@StephenChan StephenChan commented Dec 20, 2023

Cache feature vectors in the local filesystem if they were loaded from remote storage (S3 or URL).

I think this will do what it intends to, but I couldn't figure out a way to do it without committing OOP crimes somewhere (data_classes.ImageFeatures.load() in this case). I think the better way to do this would involve passing a Storage to TrainClassifierMsg instead of a DataLocation, and putting the temporary directory attribute onto that Storage. That probably involves a much larger refactor overall where the designs of Storage, DataLocation, and perhaps DataClass are reworked. I have ideas for that but I don't think I'm up for that for the remainder of the month.

Also want to make the caching optional (but on by default), so that it can be turned off in case filesystem space is a concern, as it may be for coralnet's largest sources (particularly since older sources have feature vectors about 8x in filesize).

Let me know if it'd be useful to merge this PR in the short term though.

@StephenChan
Copy link
Member Author

StephenChan commented Dec 20, 2023

Source Images Points/im Epochs Time for epoch 1 Time for each subsequent epoch
1097 1686 10 3 96s 3s
3354 1600 50 10 178s 8s
295 63263 10 10 66m41s 2m22s

The code may be ugly so far, but the results sure aren't... (remember, each subsequent epoch used to be as long as epoch 1)

@StephenChan
Copy link
Member Author

Feature caching PR

…otely.

OOP design can be better, but putting off a larger redesign for now.
…mp dir location, specify a temp dir location, or don't cache.
Also revise TrainClassifierMsg comments for the new optional arg.
@StephenChan
Copy link
Member Author

StephenChan commented Mar 16, 2024

Ready for review. Recent changes:

  • Rebased on top of recently merged PRs
  • Combined the two old commits (...since the second commit message had a typo which was bugging me)
  • Added options for the caching behavior: OS-designated temp dir location, custom temp dir location, or no caching. No caching is useful if the caching could cause disk-space problems.
  • Unit test improvement
  • Updated README and CHANGELOG

Still want to refactor adjacent classes/methods for better OOP design, but not urgent and will leave that for another time.

@StephenChan StephenChan changed the title Training: cache feature vectors for 2nd epoch onward (very WIP) Training: cache feature vectors for 2nd epoch onward Mar 16, 2024
@StephenChan StephenChan merged commit bfdc940 into main Mar 26, 2024
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants