Per package CPU optimized CFLAGS

[vizanto]

Optionally compiling certain packages optimized for your CPU, in a pure way allowing the nix store to be shared across machines would be very nice to have.

Discussion on the mailinglist:
http://lists.science.uu.nl/pipermail/nix-dev/2013-February/010604.html

Make sure -O${optimizationLevel} -march=native -mtune=native gets evaluated on the machine requesting a build while allowing build servers to do the actual compiling ( __preferLocalBuild ? )

[bluescreen303]

Danny Wilson notifications@github.com writes:

Optionally compiling certain packages optimized for your CPU, in a pure way allowing
the nix store to be shared across machines would be very nice to have.

I agree this would be nice, but I think the "price" of doing this can be
very high. Mainly because of reverse-dependencies.

As an example, think of ffmpeg. It has a lot of optional functionality
(translating into configure flags) which you can enable through
"ffmpeg.override {flag1 = true; flag2 = false;}". This has the same
impact as changing cflags, namely: it requires a rebuild from source.

Now, you have 2 options:

• install ffmpeg-with-your-flags into your user/system environment.
  Now, using "ffmpeg" from your shell will use the customized version.
  But packages on your system that depend on ffmpeg (video players,
  editing, encoding tools, perhaps your full desktop environment
  (through gstreamer), will still be using the default version.
  If this is all you need (optimizing an application that gets used
  directly), then this is cheap
• override the default ffmpeg in nixpkgs with the custom version.
  This can still be done in ~/.nixpkgs/config.nix
  Now, every package on your system will use the customized/optimized
  ffmpeg. But as a result of how nix works, this requires all these
  packages to be rebuilt from source. So if the package you want to
  optimize is a heavily-used library (like gtk+ or openssl, zlib), this
  will rebuild your entire system. This is not just a 1-time job, as
  everything you install afterwards (that depends on your optimized
  package) won't be able to use the channel's binary packages and build
  from source as well. Upgrading nixos will probably become a task that
  takes some time. And because of nix's thoroughness, this is not just
  for libraries/binaries (that are linked), but for interpreted
  shell/perl/python scripts as well (runtime or buildtime scripts), so
  optimizing bash system-wide is probably not a good idea.

There are some proposals to allow "impure patching" replacing a
dependency of a store path with a drop-in-replacement. The main usecase
would be fast security patches (for example to openssl), without having
to wait for the build farm to rebuild each and every package.
Probably this CFLAGS usecase can benefit from such a feature as well.
However, using it is impure, and there is no way to determine if a
replacement is really a 100% drop-in-replacement.

A better solution might be to have a system-wide optimization "profile",
and have the build farm (or volunteers) build stuff for certain sets of
popular profiles (core2, corei7, ...) so it's available from (custom)
channels. But as this leads to a lot of extra build-time and disk space
usage, the performance-gains would have to be worth it (not just 1 or
2%). I would be interested to see some benchmarks for a system that was
built from scratch with cpu-specific optimizations (starting as low as
glibc).

Discussion on the mailinglist:
http://lists.science.uu.nl/pipermail/nix-dev/2013-February/010604.html

—
Reply to this email directly or view it on GitHub.

[vizanto]

By making it per package, the price can be kept under control. My personal use case would be to compile an optimized webserver and database for example, not glibc or even the entire system. For heavily used processes, every extra percentage gained is pretty much a 'free' server upgrade.

However if you do end up recompiling half the system, it might be interesting to have every dependency also be optimized while you're at it. So when setting ffmpeg to optimized, have the packages depending on it be optimized as well.

Here are some (older) benchmarks comparing Gentoo optimized packages with Ubuntu 9: http://www.linux-mag.com/id/7715/2/

[aszlig]

Please see the answer from @edolstra from the ML thread you mentioned:
http://lists.science.uu.nl/pipermail/nix-dev/2013-February/010605.html

The benchmark actually came to a different conclusion than your comment, wrong URL maybe?

[bluescreen303]

On Fri, Feb 8, 2013 at 9:41 AM, Danny Wilson notifications@github.comwrote:

By making it per package, the price can be kept under control. My personal
use case would be to compile an optimized webserver and database for
example, not glibc or even the entire system. For heavily used processes,
every extra percentage gained is pretty much a 'free' server upgrade.

I see your point, but what I tried to explain is that
massive-rebuild-issues are not so much per-package/system-wide related.
They are related to whether or not you want to replace a package in nixpkgs
with an optimized version.

Judging by the syntax you proposed in the original discussion, this is what
you aim for (per system of course).
"nixpkgs.config.servers.http.nginx.optimize_cflags = true;"

This would have the effect that the expression for nginx evaluates to some
optimized version.
In stalling it, or any package depending on it, would become from-source.
This is not a problem for nginx, but it might be for a database such as
mysql/postgres.
On my (desktop) system, this would rebuild all qt/kde applications, because
qt depends on it.

The alternative is to not override nixpkgs (not use nixpkgs.config), but
just create an optimized copy.
If you install packages "the collection way" [1], or
use environment.systemPackages in configuration.nix, this would look
something like: (pkgs.mysql.override { optimize_cflags = true; }) or
(pkgs.mysql.override { stdenv = stdenv_cpu_optimizations; }). This would
give you a system with just that package optimized.

However, everything else on the system (if anything) that depends on such
a package, will be using the "default copy".
This is not a problem for an application such as nginx, which probably
doesn't have other packages depending on it, but will be for things like
mysql, which are not just an application, but provide libraries too.
You would end up with an optimized mysql server instance, while clients
such as php will still be using the non-optimized mysql client libraries on
the same system. While this might be ok, it is probably not what you
expected.

So, although I would like this ability (building an optimized package),
exposing it through an easy boolean would probably lead to users that are
less familiar with nix's workings shooting themselves in the foot.

However if you do end up recompiling half the system, it might be
interesting to have every package depending on the package you want
optimized also be optimized while you're at it. So when setting ffmpeg to
optimized, have the packages depending on it be optimized as well.

As optimizations will probably be applied through stdenv (the environment
used during package builds), this would mean that the "stdenv" input for a
package expression needs to depend on the stdenv used by all other inputs.
If any of them is stdenv_optimized, it will resolve to stdenv_optimized as
well, otherwise it needs to resolve to the default stdenv.
While this can probably be done through the callpackage mechanism (but
still be very hacky in case of optional dependencies), I think it
complicates/obscures things too much.

Here are some (older) benchmarks comparing Gentoo optimized packages with
Ubuntu 9: http://www.linux-mag.com/id/7715/2/

Nice, quite a few areas of impact, although the overall conclusion does not
spark me into using optimizations.
Also, this is comparing 32bit architecture optimizations. As a lot of extra
instructions/capabilities were added from i486 upto the core2, I would
expect optimizations to really make a difference there.
On x86_64, however, I'm pretty sure gcc already uses most of these features
by default (as there has never been an x86_64 without MMX or SSE). So
optimizations for x86_64 would only help enabling features/instructions
that exist now (on latest i7), but not on the original athlon64.

Would still be interesting to see how 64bit developed since it's early days
:)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/305#issuecomment-13281439.

[1] http://nixos.org/wiki/Howto_keep_multiple_packages_up_to_date_at_once

[vizanto]

@aszlig if you take the graphs you can see alot of packages take 5-10% less time compared to ubuntu. The written conclusion seems to ignore that. Gzip is even more dramatic, and webservers gzip all day.

[viric]

On Fri, Feb 08, 2013 at 07:48:02AM -0800, Danny Wilson wrote:

@aszlig if you take the graphs you can see alot of packages take 5-10% less time compared to ubuntu. The written conclusion seems to ignore that. Gzip is even more dramatic, and webservers gzip all day.

For what I see, the i686 gzip is close to the newer platforms (P3/Core2). It
only wins over i486. NixOS/Nixpkgs should be for i686, and if it is not, then we
should fix it.

[shlevy]

An example config.nix that should work with stdenv-updates (in principle, I haven't tested):

let
  extraCflagsFun = pkgs: import (pkgs.runCommand "cflags" {
    preferLocalBuild = true;
    __noChroot = true;
    hashChangingValue = builtins.readFile /some/system-dependent-file-that-doesn't-have-size-0 or builtins.currentTime;
  } ''
    mkdir $out
    echo "" | gcc -O3 -march=native -mtune=native -v -E - 2>&1 |grep cc1 |sed -r 's/.*? - -(.*)$/-\1/' > $out/flags
    echo "builtins.readFile ./flags" > $out/default.nix
  '';

  optimizePackageFun = pkgs: pkg: pkgs.lib.overrideDerivation pkg (attrs: {
    NIX_CFLAGS_COMPILE = "${attrs.NIX_CFLAGS_COMPILE or ""} ${extraCflagsFun pkgs};
  });
in {
  packageOverrides = pkgs:
    let
      optimizePackage = optimizePackageFun pkgs;
    in {
      # This one doesn't override the real package, so anything that depends on vim gets
      # an unoptimized version
      customVim = optimizePackage pkgs.vim;

      # Super-fast command line history for everyone!
      readline = optimizePackage pkgs.readline;
    };
}

[shlevy]

Not sure why I mentioned stdenv-updates, the above should work just fine in master too.

[shlevy]

Does my example config.nix solve the issue for those who care?

[shlevy]

Please feel free to reopen if there's still an issue here.

[vizanto]

I just re-read the discussion, can I also optimize (override?) the dependencies of the to-be optimized package with this function?

Will test this out soon with our custom packages :-)

[shlevy]

Hmm that's harder to do in general, but for the majority of packages (i.e. those that use makeOverridable, which is called as part of callPackage) you could make an overridden stdenv and then use deepOverride on the package you want to override the dependencies of.