Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

icestorm: parallel builds, use pypy3 #53721

Merged

Conversation

Projects
None yet
2 participants
@thoughtpolice
Copy link
Member

commented Jan 10, 2019

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nox --run "nox-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Assured whether relevant documentation is up to date
  • Fits CONTRIBUTING.md.

@thoughtpolice thoughtpolice force-pushed the thoughtpolice:nixpkgs/icestorm-performance branch from 672fd58 to a30f335 Jan 10, 2019

thoughtpolice added some commits Jan 10, 2019

icestorm: improve x86 build/runtime perf with pypy
PyPy3 offers tremendous speedups for IceStorm tools written in Python,
including tools used at compile-time to generate the chip databases, and
runtime tools distributed to users, such as icebox_vlog.

For example, on my ThreadRipper 1950X, build times for IceStorm
consistently go from 2m30s -> 1m30s with this change, a 40% improvement,
simply due to improvements in raw CPU efficiency. (This is also worsened
by the fact the build is currently serial, but that can easily be fixed
anyway.)

On top of that, tools distributed to users are also now run using PyPy.
Utilities such as icebox_vlog are useful for post-bitstream testing, for
instance, and also are improved due to improved CPU efficiency as well.
For example, when "decompiling" an ICE40 bitstream for HX8K devices,
containing a synthesized copy of PicoRV32 (from the NextPNR demos), the
runtime of icebox_vlog is cut from 25 seconds to 9 seconds consistently
with this change alone.

Normally, picking a Python interpreter outright for Python-based code is
a "bad idea", but in the case of IceStorm it should be perfectly safe,
and an excellent improvement for users. There are a few reasons for
this:

  - IceStorm uses pure Python 3 and nothing else. There are no
requirements for any 3rd party packages, which might cause annoying
incompatibilities, and PyPy has historically shown very strong core
Python compatibility.

  - IceStorm is NOT a set of Python libraries, it is a set of tools,
some of which, coincidentally, are written in Python. It is (normally)
bad form to fix libraries to certain interpreters versions if the reason
strictly isn't "it doesn't work/isn't compatible". That is not the case
here. These tools may later be used by other programs, such as NextPNR,
but the Python interpreter is ultimately not that important in quesion
for the user. In this sense, there is almost no downside to picking
PyPy explicitly if it offers far better performance.

(Point 2 is not actually strictly true; there are some distributed .py
files that you can import from but they are basically just static
classes that are imported by tools like nextpnr; this is expected.)

Because of this, users should see very little change except better
performance for IceStorm tools on their machines.

Note that PyPy is not supported on aarch64 -- this only applies to
x86_64 machines.

Signed-off-by: Austin Seipp <aseipp@pobox.com>
icestorm: enableParallelBuilding = true
With the previous PyPy3 change, this reduces the compile time from
~1m30s to roughly 36s (compared to the original, serial, Python 3 build
time of 2:30s).

Signed-off-by: Austin Seipp <aseipp@pobox.com>

@thoughtpolice thoughtpolice force-pushed the thoughtpolice:nixpkgs/icestorm-performance branch from a30f335 to 88ede21 Jan 11, 2019

@thoughtpolice thoughtpolice merged commit 61e57a8 into NixOS:master Jan 12, 2019

12 of 13 checks passed

nix-build -A icestorm --argstr system x86_64-darwin Build Results
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
nix-build -A icestorm --argstr system aarch64-linux Build Results
Details
nix-build -A icestorm --argstr system x86_64-linux Build Results
Details

@thoughtpolice thoughtpolice deleted the thoughtpolice:nixpkgs/icestorm-performance branch Jan 12, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.