Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

per device worker #205

Merged
merged 40 commits into from
Mar 1, 2023
Merged

per device worker #205

merged 40 commits into from
Mar 1, 2023

Commits on Feb 26, 2023

  1. begin switching to per-device torch mp workers

    @ssube
    ssube committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    e46a1e5 View commit details
    Browse the repository at this point in the history

  2. background workers, logger

    @ssube
    ssube committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    f898de8 View commit details
    Browse the repository at this point in the history

  3. wire up worker jobs

    @ssube
    ssube committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    943281f View commit details
    Browse the repository at this point in the history

  4. feat(api): remove Flask app from global scope

    @ssube
    ssube committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    06c74a7 View commit details
    Browse the repository at this point in the history

  5. rejoin worker pool

    @ssube
    ssube committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    6998e87 View commit details
    Browse the repository at this point in the history

  6. make logger start up well

    @ssube
    ssube committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    d765a6f View commit details
    Browse the repository at this point in the history

  7. lock per worker, torch before ORT

    @ssube
    ssube committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    e1d0ad5 View commit details
    Browse the repository at this point in the history

  8. apply patches within workers

    @ssube
    ssube committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    f115326 View commit details
    Browse the repository at this point in the history

  9. update progress and finished flag from worker

    @ssube
    ssube committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    e0737e9 View commit details
    Browse the repository at this point in the history

  10. recycle worker pool after 10 jobs

    @ssube
    ssube committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    6502e1e View commit details
    Browse the repository at this point in the history

  11. set process titles, terminate workers

    @ssube
    ssube committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    b880b7a View commit details
    Browse the repository at this point in the history

  12. lint all the new stuff

    @ssube
    ssube committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    584dddb View commit details
    Browse the repository at this point in the history

  13. re-implement cancellation

    @ssube
    ssube committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    d1961af View commit details
    Browse the repository at this point in the history

  14. clear worker flags between jobs, attempt to record finished jobs again

    @ssube
    ssube committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    85118d1 View commit details
    Browse the repository at this point in the history

  15. fix imports, lint

    @ssube
    ssube committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    b931da1 View commit details
    Browse the repository at this point in the history

  16. initialize list of finished jobs

    @ssube
    ssube committed Feb 26, 2023
    Configuration menu
    Copy the full SHA
    eb82e73 View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2023

  1. track started and finished jobs

    @ssube
    ssube committed Feb 27, 2023
    Configuration menu
    Copy the full SHA
    525ee24 View commit details
    Browse the repository at this point in the history

  2. fix finished flag

    @ssube
    ssube committed Feb 27, 2023
    Configuration menu
    Copy the full SHA
    401ee20 View commit details
    Browse the repository at this point in the history

  3. use progress queue

    @ssube
    ssube committed Feb 27, 2023
    Configuration menu
    Copy the full SHA
    a37d1a4 View commit details
    Browse the repository at this point in the history

  4. always put progress in active jobs

    @ssube
    ssube committed Feb 27, 2023
    Configuration menu
    Copy the full SHA
    1339593 View commit details
    Browse the repository at this point in the history

  5. run logger in a thread, clean up status

    @ssube
    ssube committed Feb 27, 2023
    Configuration menu
    Copy the full SHA
    66a20e6 View commit details
    Browse the repository at this point in the history

  6. join all threads

    @ssube
    ssube committed Feb 27, 2023
    Configuration menu
    Copy the full SHA
    2327b24 View commit details
    Browse the repository at this point in the history

  7. typo

    @ssube
    ssube committed Feb 27, 2023
    Configuration menu
    Copy the full SHA
    113ad05 View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2023

  1. error handling in all threads

    @ssube
    ssube committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    06f06f5 View commit details
    Browse the repository at this point in the history

  2. fix Windows entrypoint

    @ssube
    ssube committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    61373d5 View commit details
    Browse the repository at this point in the history

  3. consistently pass job key to workers

    @ssube
    ssube committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    0793b61 View commit details
    Browse the repository at this point in the history

  4. set queue timeouts

    @ssube
    ssube committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    1367592 View commit details
    Browse the repository at this point in the history

  5. handle empty errors

    @ssube
    ssube committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    953e5ab View commit details
    Browse the repository at this point in the history

  6. quit workers on keyboard signal

    @ssube
    ssube committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    988088d View commit details
    Browse the repository at this point in the history

  7. more logging around shutdown, close queues

    @ssube
    ssube committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    da6ae5d View commit details
    Browse the repository at this point in the history

  8. directly rejoin pool

    @ssube
    ssube committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    f7f438e View commit details
    Browse the repository at this point in the history

  9. add value error handling

    @ssube
    ssube committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    1ce98ac View commit details
    Browse the repository at this point in the history

  10. remove pending queues after joining

    @ssube
    ssube committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    7e0ccdb View commit details
    Browse the repository at this point in the history

  11. remove task done

    @ssube
    ssube committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    4ae3d9c View commit details
    Browse the repository at this point in the history

  12. some pending queue logging

    @ssube
    ssube committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    cad0d37 View commit details
    Browse the repository at this point in the history

  13. daemonize queue collectors

    @ssube
    ssube committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    0011f07 View commit details
    Browse the repository at this point in the history

  14. avoid terminating workers because it breaks their queues

    @ssube
    ssube committed Feb 28, 2023
    Configuration menu
    Copy the full SHA
    c95ac1f View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2023

  1. name threads, max queues, type/lint fixes

    @ssube
    ssube committed Mar 1, 2023
    Configuration menu
    Copy the full SHA
    c99aa67 View commit details
    Browse the repository at this point in the history

  2. fix(api): sanitize filenames in user input

    @ssube
    ssube committed Mar 1, 2023
    Configuration menu
    Copy the full SHA
    12fb7f5 View commit details
    Browse the repository at this point in the history

  3. apply lint

    @ssube
    ssube committed Mar 1, 2023
    Configuration menu
    Copy the full SHA
    1f9efb4 View commit details
    Browse the repository at this point in the history