Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable fully threaded IO for installs #1876

Merged
merged 7 commits into from Jun 15, 2019
Merged

Commits on Jun 6, 2019

  1. Enable threaded closing on all platforms.

    Set RUSTUP_CLOSE_THREADS=disabled to force single threaded IO, or
    to a specific number if desired for testing/tuning.
    
    This may improve rust-lang#1867, but has no impact on rust-lang#1866 due to the coarse
    lock around the fd-handle table inside WSL.
    rbtcollins committed Jun 6, 2019
    Copy the full SHA
    6e6f44e View commit details
    Browse the repository at this point in the history
  2. Move unpacking into rustup entirely.

    Generalises the threaded IO closing to be fully threaded disk IO.
    rbtcollins committed Jun 6, 2019
    Copy the full SHA
    0ddcf1c View commit details
    Browse the repository at this point in the history
  3. Trace (most) installation disk operations

    Adds a new environment variable to control tracing. Tracing is
    manual and can be expanded. We can probably get better systemic
    tracing from a suite of platform specific tools over time, but for
    now this was low hanging fruit that allows pretty good insight
    without too much code intrusion. Runtime overhead when disabled
    is extremely low (I can unpack rust-docs on Linux in 1s).
    rbtcollins committed Jun 6, 2019
    Copy the full SHA
    e9b5e24 View commit details
    Browse the repository at this point in the history
  4. Eliminate post-install directory-walk

    This was only needed because we were not controlling the permissions
    accurately during unpack, which can now be corrected.
    rbtcollins committed Jun 6, 2019
    Copy the full SHA
    b146976 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2019

  1. Thread dir creation as well

    When directories complete, start writing the files/dirs within
    that directory that have been decompressed already.
    
    Avoids a stat() + create_dir_all() in the main thread permitting
    more concurrent IO dispatch in exchange for memory pressure.
    rbtcollins committed Jun 14, 2019
    Copy the full SHA
    d614d60 View commit details
    Browse the repository at this point in the history
  2. Limit RAM use during unpacking.

    Heavily deferred directories can lead to very large amounts of RAM
    holding pending IOs, and under low memory limits this caused failures.
    
    Reduce the stack size for IO worker threads to 1MB, as the default
    of 8MB on Linux is wasteful for IO worker threads.
    rbtcollins committed Jun 14, 2019
    Copy the full SHA
    517a477 View commit details
    Browse the repository at this point in the history
  3. Review feedback.

    rbtcollins committed Jun 14, 2019
    Copy the full SHA
    35dd1ed View commit details
    Browse the repository at this point in the history