Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Faster image load #189

Merged
merged 10 commits into from
Oct 11, 2023
Merged

Faster image load #189

merged 10 commits into from
Oct 11, 2023

Commits on Oct 10, 2023

  1. Refactor progress bar

    @hackartisan
    hackartisan committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    b4d3c7e View commit details
    Browse the repository at this point in the history

  2. Refactor CardImageLoadingService to use aws ls per disk 

    instead of aws ls per database object

closes #188
    @hackartisan
    hackartisan committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    95b4009 View commit details
    Browse the repository at this point in the history

  3. Use percentage progress bar with ETA, and log each disk number

    @hackartisan
    hackartisan committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    0e25dd9 View commit details
    Browse the repository at this point in the history

  4. Update readme with new image load time estimate

    @hackartisan
    hackartisan committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    5e9a5fe View commit details
    Browse the repository at this point in the history

  5. Don't fetch disk 11 with disk 1. 

    also fix spec nesting
    @hackartisan
    hackartisan committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    500cb88 View commit details
    Browse the repository at this point in the history

  6. Fix rubocop

    @hackartisan
    hackartisan committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    73c3052 View commit details
    Browse the repository at this point in the history

  7. simplify s3 command

    @hackartisan
    hackartisan committed Oct 10, 2023
    Configuration menu
    Copy the full SHA
    6d90cdc View commit details
    Browse the repository at this point in the history

Commits on Oct 11, 2023

  1. Use async to speed up db writes

    @hackartisan
    hackartisan committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    27223df View commit details
    Browse the repository at this point in the history

  2. Do 8 concurrent tasks instead of 10 

    This splits the 22 sets of images up more evenly, and uses a bit fewer
concurrent resources. Previous iteration overwhelmed the staging machine
into a memory allocation crash. Resources have also been increased, and
now there are 8 cores, so 8 seems good for that reason, too.
    @hackartisan
    hackartisan committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    2b0e132 View commit details
    Browse the repository at this point in the history

  3. Reduce resources used on 2nd async with a semaphore

    @hackartisan
    hackartisan committed Oct 11, 2023
    Configuration menu
    Copy the full SHA
    794497f View commit details
    Browse the repository at this point in the history