Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

optimize derivation parsing #9673

Merged
merged 5 commits into from
Dec 31, 2023
Merged

optimize derivation parsing #9673

merged 5 commits into from
Dec 31, 2023

Commits on Dec 29, 2023

  1. don't use istreams in hot paths

    istream sentry objects are very expensive for single-character
    operations, and since we don't configure exception masks for the
    istreams used here they don't even do anything. all we need is
    end-of-string checks and an advancing position in an immutable memory
    buffer, both of which can be had for much cheaper than istreams allow.
    
    the effect of this change is most apparent on empty stores.
    
    before:
    
    Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
      Time (mean ± σ):      7.167 s ±  0.013 s    [User: 5.528 s, System: 1.431 s]
      Range (min … max):    7.147 s …  7.182 s    10 runs
    
    after:
    
    Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
      Time (mean ± σ):      6.963 s ±  0.011 s    [User: 5.330 s, System: 1.421 s]
      Range (min … max):    6.943 s …  6.974 s    10 runs
    pennae committed Dec 29, 2023
    Configuration menu
    Copy the full SHA
    99a691c View commit details
    Browse the repository at this point in the history
  2. malloc/memset even less

    more buffers that can be uninitialized and on the stack. small
    difference, but still worth doing.
    
    before:
    
    Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
      Time (mean ± σ):      6.963 s ±  0.011 s    [User: 5.330 s, System: 1.421 s]
      Range (min … max):    6.943 s …  6.974 s    10 runs
    
    after:
    
    Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
      Time (mean ± σ):      6.952 s ±  0.015 s    [User: 5.294 s, System: 1.452 s]
      Range (min … max):    6.926 s …  6.974 s    10 runs
    pennae committed Dec 29, 2023
    Configuration menu
    Copy the full SHA
    2cfc4ac View commit details
    Browse the repository at this point in the history

Commits on Dec 30, 2023

  1. optimize derivation string parsing

    a bunch of derivation strings contain no escape sequences. we can
    optimize for this fact by first scanning for the end of a derivation
    string and simply returning the contents unmodified if no escape
    sequences were found. to make this even more efficient we can also use
    BackedStringViews to avoid copies, avoiding heap allocations for
    transient data.
    
    before:
    
    Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
      Time (mean ± σ):      6.952 s ±  0.015 s    [User: 5.294 s, System: 1.452 s]
      Range (min … max):    6.926 s …  6.974 s    10 runs
    
    after:
    
    Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
      Time (mean ± σ):      6.907 s ±  0.012 s    [User: 5.272 s, System: 1.429 s]
      Range (min … max):    6.893 s …  6.926 s    10 runs
    pennae committed Dec 30, 2023
    Configuration menu
    Copy the full SHA
    79d3d41 View commit details
    Browse the repository at this point in the history
  2. use translation table for drv string parsing

    the table is very small compared to cache sizes and a single indexed
    load is faster than three comparisons.
    
    before:
    
    Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
      Time (mean ± σ):      6.907 s ±  0.012 s    [User: 5.272 s, System: 1.429 s]
      Range (min … max):    6.893 s …  6.926 s    10 runs
    
    after:
    
    Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
      Time (mean ± σ):      6.883 s ±  0.016 s    [User: 5.250 s, System: 1.424 s]
      Range (min … max):    6.860 s …  6.905 s    10 runs
    pennae committed Dec 30, 2023
    Configuration menu
    Copy the full SHA
    02c64ab View commit details
    Browse the repository at this point in the history
  3. reduce copies during drv parsing

    many paths need not be heap-allocated, and derivation env name/valye
    pairs can be moved into the map.
    
    before:
    
    Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
      Time (mean ± σ):      6.883 s ±  0.016 s    [User: 5.250 s, System: 1.424 s]
      Range (min … max):    6.860 s …  6.905 s    10 runs
    
    after:
    
    Benchmark 1: nix eval --raw --impure --expr 'with import <nixpkgs/nixos> {}; system'
      Time (mean ± σ):      6.868 s ±  0.027 s    [User: 5.194 s, System: 1.466 s]
      Range (min … max):    6.828 s …  6.913 s    10 runs
    pennae committed Dec 30, 2023
    Configuration menu
    Copy the full SHA
    c62686a View commit details
    Browse the repository at this point in the history