Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Limit concurrency #329

Draft
wants to merge 10 commits into
base: main
Choose a base branch
from
Draft

Limit concurrency #329

wants to merge 10 commits into from

Commits on Nov 9, 2021

  1. Add max_concurrency option to limit concurrency

    This adds a new keyword to AzureBlobFileSystem to limit the number of
    concurrent connectiouns. See pangeo-forge/pangeo-forge-recipes#227 (comment)
    for some motivation. In that situation, we had a single FileSystem
    instance that was generating many concurrent write requests through
    `.pipe`. So many, that we were seeing memory issues from creating all
    the BlobClient connections simultaneously.
    
    This adds an asyncio.Semaphore instance to the AzureBlobFilesSytem that
    controls the number of concurrent BlobClient connections. The default of
    None is backwards-compatible (no limit)
    Tom Augspurger committed Nov 9, 2021
    Configuration menu
    Copy the full SHA
    6cc062d View commit details
    Browse the repository at this point in the history
  2. Update docs

    Tom Augspurger committed Nov 9, 2021
    Configuration menu
    Copy the full SHA
    14c3fe1 View commit details
    Browse the repository at this point in the history
  3. compat

    Tom Augspurger committed Nov 9, 2021
    Configuration menu
    Copy the full SHA
    1b82b09 View commit details
    Browse the repository at this point in the history

Commits on Jul 23, 2022

  1. Fixing merge conflicts

    hayesgb committed Jul 23, 2022
    Configuration menu
    Copy the full SHA
    d4631c1 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3efa3dd View commit details
    Browse the repository at this point in the history
  3. Fixing merge conflicts

    hayesgb committed Jul 23, 2022
    Configuration menu
    Copy the full SHA
    972ea49 View commit details
    Browse the repository at this point in the history
  4. Linting

    hayesgb committed Jul 23, 2022
    Configuration menu
    Copy the full SHA
    61c7867 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2022

  1. Removing print statements

    hayesgb committed Jul 25, 2022
    Configuration menu
    Copy the full SHA
    2966f63 View commit details
    Browse the repository at this point in the history
  2. Linting

    hayesgb committed Jul 25, 2022
    Configuration menu
    Copy the full SHA
    ef14b6f View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    a787e39 View commit details
    Browse the repository at this point in the history