Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve parallel read #74

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Improve parallel read #74

wants to merge 9 commits into from

Commits on Jan 31, 2023

  1. improve _ParallelRead not reading the whole folder into memory before… 

    … starting to work
    @leo-schick
    leo-schick committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    8068542 View commit details
    Browse the repository at this point in the history

  2. fix typing

    @leo-schick
    leo-schick committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    0a4e549 View commit details
    Browse the repository at this point in the history

  3. small fixes

    @leo-schick
    leo-schick committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    4f9e00d View commit details
    Browse the repository at this point in the history

  4. limit command_queue to limit memory usage

    @leo-schick
    leo-schick committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    eba32f6 View commit details
    Browse the repository at this point in the history

  5. do not add empty command chains into the queue

    @leo-schick
    leo-schick committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    3e9be23 View commit details
    Browse the repository at this point in the history

  6. fix endless loop when command queue is full from FeedWorkerProcess an… 

    …d a Worker node failed

add basic unit testing for FeedWorkerProcess logic
add unit test for when command queue is full
    @leo-schick
    leo-schick committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    0c5c5ad View commit details
    Browse the repository at this point in the history

  7. restructuring code, fixing tests

    @leo-schick
    leo-schick committed Jan 31, 2023
    Configuration menu
    Copy the full SHA
    e9a78a8 View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2023

  1. fix crash when the FeedWorkersProcess exists before the Worker proces… 

    …s has started
    @leo-schick
    leo-schick committed Feb 22, 2023
    Configuration menu
    Copy the full SHA
    ad0bdf6 View commit details
    Browse the repository at this point in the history

Commits on Feb 24, 2023

  1. fix endless running worker nodes when feed worker process fails

    @leo-schick
    leo-schick committed Feb 24, 2023
    Configuration menu
    Copy the full SHA
    3f17514 View commit details
    Browse the repository at this point in the history