Merging x-updates

[vcunat]

I think we should just stabilize x-updates and merge it soon (within a few weeks).

Currently contained biggest changes:

• libpng16 is the default (If the bugfix release comes within the time I think we should update .1 -> .2.)
• libjpeg-turbo is the default, see libjpeg-turbo #299 for details.
• mesa 9, which also seems to fix some problems Teeworlds segfaults #447
• gtk3 gets updated 3.2.x -> 3.8.0, also some other GNOME 3.8 stuff (mainly basic libraries, e.g. new glib, etc.)
• xorg-server branch 1.13, xorg updates #471
• many smaller library updates
• font rendering changes substantially: the libraries got updated, but most should be caused by the Infinality patches 610796d. IMO the fonts look/read significantly better now. Also the configurability should be much better. It costs us some speed but this shouldn't be serious due to font caching. This change still needs some investigation to check again there are no problems with this (I'll do this before merging, if noone wants to do it).

Other things we may potentially want to include:

• gstreamer-1 Merging gstreamer-1.0 branch? #479, probably still needs to be reviewed
• cups seems old to me, and all gtk2/3, qt4 depend on it (I'm probably not the right person to handle this)

I'm running the whole system on current x-updates now, so at least the common parts and nvidia driver get some testing. I don't much know e.g. what are the best choices for GFX drivers for Intel and AMD/ATI...

Any thoughts?

[vcunat]

I forgot to mention that the number of broken builds (w.r.t. master) is very low; my estimate is a few hundreds at most (across all Hydra platforms). Current Hydra stats are confusing in this respect, as we have accumulated many negatively cached transient errors on the branch (now on master as well).

[viric]

I've a local branch that adds gnunet-gtk, that depends on gtk 3.8. I'll try to
merge it to x-updates. git gave me weird troubles, when I merged it locally to
x-updates though; I've to check it better.

I think it would be nice to decide clear instructions on how to test this branch
exactly previous to the merge, for most people to participate.

[7c6f434c]

When LibreOffice was last working in x-updates, I used x-updates for a while; I'd say that font rendering was significantly worse than in trunk. Can I just set some environment variables to get nice thin lines back?

I think Intel drivers are OK. What about LibreOffice?

[vcunat]

• Ah, yes, I should mention that to if you want to get substitutes from Hydra, you probably have to run sudo nix-pull http://hydra.nixos.org/jobset/nixpkgs/xorg-test/channel/latest/MANIFEST
• LibreOffice: it would be nice if someone updated it, but probably nothing depends on it, so it can be done anytime in master.
• Fonts: well, yes, IMO one of the main things to think about is what default font settings we want. Configurability should work through at least env-vars and /etc/*. I think the options can be found from the homepage http://www.infinality.net/blog/infinality-freetype-patches/ .

@viric: I edited your comments, github doesn't seem to handle citations from e-mails well, it wasn't easy to distinguish who wrote what.

[vcunat]

There seems to be a problem in new glib+ld-linux https://bugs.archlinux.org/task/34630#comment108896 . AFAIK it probably causes:

• skype crashes and
• occasional xfce4session crashes.

There's a probable work-around which I'm testing now. Also, in a few days the guys working on this may find out more, we'll see...

[vcunat]

These problems should be fixed now. The work-around from Arch also seems to fix the xfce4session crashes (I'm running it for a few days without a single problem).

[vcunat]

I just pushed a mesa bugfix update. I suppose we should merge in a few days, after Hydra builds the branch.

I expect no more feature changes now (unless someones steps in), I'll only look into the Infinality stuff and default font rendering configuration.

Edit: more info for anyone interested: http://google-opensource.blogspot.cz/2013/05/got-cff.html

[vcunat]

Hmm, it seems that font rendering might noticeably change and improve soon, again. http://git.savannah.gnu.org/cgit/freetype/freetype2.git/tree/docs/CHANGES But not for this x-updates merge, it's newly merged in freetype and still in beta now.

[vcunat]

I was testing a few bugfix updates I committed and they seem fine. Now there's only the issue with default font configuration:

• Up to now there was nothing set in environment, so it should behave as freetype defaults (as without infinality). @viric: I suppose this default has changed when updating freetype in x-updates.
• There's a fine-grained control over the appearance of fonts through the environment. Detailed description is in http://www.infinality.net/fedora/linux/zips/freetype-infinality-2.4.11-20130104_04-x86_64.tar.bz2 in file infinality-settings.sh. There are also some example combinations that should simulate particular systems (like Win7, Ubuntu, IPad) and also a compromise default profile that should please most people (and there are more aggressive profiles). IMHO it does look better than the freetype's default, but that's quite a personal thing.
• @viric: for lighter appearance you probably want to increase INFINALITY_FT_BRIGHTNESS to something like 80 (it's probably best to take one of the profiles and only update this variable). Or you may want to switch to fonts that were designed lighter (some have light variants, e.g. DejaVu Sans has ExtraLight).
• I suggest we take this default environment and add it into the system-wide profile, probably through nixos modules/config/fonts.nix (I haven't yet verified it can be done in this file).

The default font configuration belongs into nixos, so I would merge the branch as it is now. I believe almost all build errors relative to trunk are transient. Certainly, racket is an exception, but I failed to fix this one http://hydra.nixos.org/job/nixpkgs/xorg-test/racket.x86_64-linux

I'm giving you additional time to react until wednesday around noon (European time) before I merge to master. IMHO we've discussed all this for a long enough time (except for font configuration, but that's a nixos issue, not nixpkgs).

[edolstra]

Is infinality considered stable? Or put differently, why doesn't upstream freetype have this patch? Do other distributions use it?

[vcunat]

@edolstra: Well, the part adding truetype subpixel rendering is integrated into released freetype (as of version in x-updates). The other part that's added here won't be upstreamed soon and it contains many configurable tweaks to rendering. Yes, it's only considered beta and in distributions it's added as optional or unofficial AFAIK. However, with the environment clear the code paths aren't even used and the library should behave as released.

When I think of it again, I would probably leave the environment clear by default, so user get the "upstream settings" (I believe at least Ubuntu has their own tweaks to font rendering). This way users can still easily use infinality just by changing their profile (without recompiling everything). However, AFAIK most people find the upstream defaults not good-looking.

I think the configurability is the main advantage, because there just isn't one true way of font rendering.

[edolstra]

Ok, sounds good!

[bluescreen303]

Can we have an easy nixos option to set these environment variables for good-looking default configs? :)

[vcunat]

@bluescreen303: I suppose we can provide this... only there's a question which settings to give there to choose from. The shell script in the patchset provides around seven pre-defined combinations, and you can define your own, of course. And it's not always better, too, e.g. on http://dl.acm.org/citation.cfm?id=1292541 I find the headings very much too bold with infinality's "default" compromise (although the rest seems thin enough).

[edolstra]

I just noticed some extreme bloat in x-updates. For instance, the closure of the geeqie package went from 461 MiB to 851 MiB, which is really bad IMHO. This bloat seems to be due to the dependencies on llvm (166 MiB) and mesa-noglu (an insane 261 MiB). Can we do something about that? If not, I'd prefer reverting to the old mesa for the time being.

[vcunat]

I'll look into it, but I believe we've already discussed that llvm is essential for mesa now (but maybe we can clean some parts).

[edolstra]

Ok, then we should revert to Mesa 8 for now, or depend on a Mesa 9 that doesn't build all (or any) drivers. A full Mesa 9 can be provided on NixOS via LD_LIBRARY_PATH (like we do for the proprietary NVIDIA driver).

[vcunat]

Oh yes, I believe separating the drivers should do the trick. We've probably been going the other way until now for mesa, but in future this probably isn't going to get much smaller.

[edolstra]

Also wondering how Mesa managed to go from 50 to 261 MiB. Apparently every driver has grown by 16 MiB or so:

$ l /nix/store/82yrdifgdbq10lmc6z6l9w92ivq0hr6p-mesa-8.0.4/lib/dri/i915_dri.so 
-r-xr-xr-x 1 root root 4.7M Jan  1  1970 /nix/store/82yrdifgdbq10lmc6z6l9w92ivq0hr6p-mesa-8.0.4/lib/dri/i915_dri.so
$ l /nix/store/gk60dl9bd6gcdrm0rmyljl03r5pw4n42-mesa-noglu-9.1.2/lib/dri/i915_dri.so
-r-xr-xr-x 1 root root 20M Jan  1  1970 /nix/store/gk60dl9bd6gcdrm0rmyljl03r5pw4n42-mesa-noglu-9.1.2/lib/dri/i915_dri.so

[vcunat]

The mesa libs aren't stripped :-D

[edolstra]

You sure? I tried strip -S on i915_dri.so and it didn't do anything...

[vcunat]

That's what I tried first :-) But:

20M     /nix/store/jvd567ix5qcswj2kc3kbb1mlsch1j210-mesa-noglu-9.1.2/lib/dri/i915_dri.so
8.5M    i915_dri-test.so

[vcunat]

After plain strip -S -o i915_dri-test.so.

[vcunat]

Continue on IRC?

[vcunat]

Well, after much testing and discussion, my WIP is here https://github.com/vcunat/nixpkgs/tree/p/mesa

Current status:

• driver-less mesa has 7 MB and doesn't need LLVM :-)
• size of llvm got down to ~80 MB (due to dynamic linking, but that seems to cause problems elsewhere)
• the mesa drivers have 200 MB together

[vcunat]

Ad llvm: I believe the only runtime dependency for most packages using LLVM should be just libllvm, which is around 25 MB, so it could be split from the rest if we really wanted to save more space there...

[vcunat]

It seems that now this only breaks clang, I'll try to fix it. But please, someone test the mesa drivers. They got split into mesa_drivers (all at once ATM, around 200 MB + 80 MB LLVM). Something will be needed in NixOS to make the server find the drivers, probably similar as for nvidia.

I rebuilt my system with the updates, and all including 3D games works fine (without NixOS change, on nvidia proprietary driver).

I suppose we really want to resolve this quickly, as we're directly on master. Reverting to mesa-8 would be an option, but I believe it's better to step forward: do this split, quickly test it, and commit it.

[vcunat]

I forgot to mention that I setup the mesa driver search path to /run/current-system/sw/lib/dri/ so I hope it should be enough to add mesa_drivers to the system packages and they should be found.

[edolstra]

Is it possible to revert the x-updates merge in master? That way we don't need to stress out :-)

However I'm worried that Git will get horribly confused if you revert a merge (in particular if you later sync master into x-updates...).

[vcunat]

@jcumming: and that is needed? I did check linking of all executables by ldd (before committing).

[vcunat]

After discussing this on IRC, @Phreedom lead me to the fact that it was already OK. NixOS sets /run/opengl-driver{,-32}/lib into global $LD_LIBRARY_PATH, and there's also the driver-specific libGL (apps are compiled against the version from mesa, as in all distros we rely on the fact that drivers provide an ABI-compatible version).

[jcumming]

so:

1. Source packages: link against mesa's libGL.so.1, include ${mesa}/lib in RPATH?
2. Binary packages: patchelf RPATH to ${mesa}/lib?

nixos will set LD_LIBRARY_PATH so that optimized drivers will get used where possible, otherwise will fall back to mesa's libGL.so.1?

If so, then I should fix a couple of my binary packages for 2.

[vcunat]

Yes, (1) and (3) is certain and has not changed in x-updates. libGL is overridden by the current opengl driver (/run/opengl-driver{,-32}). We have LD_LIBRARY_PATH=/run/opengl-driver/lib:/run/opengl-driver-32/lib.

Now if you're running one of mesa drivers, there's no /run/opengl-driver{,-32}/lib/libGL_. This has changed now, so we can either do (2) or add link from mesa_drivers to mesa's libGL so we always have some version on /run/opengl-driver_ (seems more consistent anyway).

How did you do (2) up to now? Left without RPATH to mesa? Don't they sometimes need libGLU? This one got split away from mesa upstream, but now we have it in the combined "mesa" attribute (original is mesa_noglu).

[bluescreen303]

All is working well for me, I love the fontzies :)

Anyway, it seems xf86-video-vmware-13.0.1 is broken though.
Can't find xatracker.

On Tue, May 21, 2013 at 8:08 AM, Vladimír Čunát notifications@github.comwrote:

Yes, (1) and (3) is certain and has not changed in x-updates. libGL is
overridden by the current opengl driver (/run/opengl-driver{,-32}). We have
LD_LIBRARY_PATH=/run/opengl-driver/lib:/run/opengl-driver-32/lib.

Now if you're running one of mesa drivers, there's no
/run/opengl-driver{,-32}/lib/libGL_. This has changed now, so we can either
do (2) or add link from mesa_drivers to mesa's libGL so we always have some
version on /run/opengl-driver_ (seems more consistent anyway).

How did you do (2) up to now? Left without RPATH to mesa? Don't they
sometimes need libGLU? This one got split away from mesa upstream, but now
we have it in the combined "mesa" attribute (original is mesa_noglu).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/490#issuecomment-18190896
.

[bluescreen303]

Isn't the llvm dependency only necessary for the software-rendering fallback?
Or do other (DRI) drivers use it as well?

[vcunat]

@bluescreen303: I don't see into internals, but I think I read it's also used for JIT optimizing shaders for the hardware you're running.

xatracker also seemed to depend on llvm, so it's in drivers. I'll fix the vmware driver to find it there. I'm also currently testing a bugfix-only mesa update (9.1.2 -> 9.1.3)... I suppose it should be included, I'll push it soon.

[vcunat]

Well, I suppose we want to include the new X security fixes before merging to master. The problem is that the changes are still unreleased or in unstable versions. It's announced that they want to release the first stable "secure" version of libX11 during the first week of June. I believe it's worth waiting for it (although I hate that we're putting off the merge again and again).

[security announcement] http://www.x.org/wiki/Development/Security/Advisory-2013-05-23
[libX11 stabilization] http://lists.x.org/archives/xorg-announce/2013-May/002221.html

[edolstra]

I saw that Debian has released updates for the affected packages. I guess they've backported the security fixes. Maybe we can borrow their patches.

[vcunat]

Yes, we could do that, although it will need quite a lot of temporary code, probably in overrides.nix. I'm a bit overloaded with work this week, but if anyone hurries with this, he/she is welcome to do it.

[vcunat]

Progress report:

• X has still made no release of three insecure libs, so I took their git versions and the released rest 619b024.
• @iElectric noticed another bugfix python update, so that one was also pushed.
• Hydra will take some time to build this, but we can merge to master soon afterwards (IMO in a few days, unless some unexpected issue occurs).
• Together with it we will probably also merge https://github.com/NixOS/nixpkgs/branches/stdenv-fixes for some security stdenv fixes, so say if you know about any non-disruptive change that could be included (when we're rebuilding all anyway).

[domenkozar]

It's almost done, there are some package failures (not that important ones)

[edolstra]

I finally got around to testing this on my Asus eeePC, which has an Intel 915 graphics card. (All my other systems have NVIDIA.) Everything (meaning Xfce) seems to work fine. I'm +1 on merging this (together with the stdenv-fixes branch).

[domenkozar]

Groff doesn't build on FreeBSD (and all reverse dependencies such as python), if anyone cares: http://hydra.nixos.org/build/5257601

[vcunat]

I believe it can be merged anytime now. groff does cause many failures, but I don't even have a possibility to test it. I also wanted to enable better fonts on XUL-based apps, but I'm hitting https://bugzil.la/856419 and it doesn't do subpixel rendering anyway...

[peti]

groff compiles just fine for me:

[simons@freebsd-8 ~/nixpkgs]$ nix-build -o /tmp/groff ~/nixpkgs -A groff 
/nix/store/jaf2pjl9acb5p30180iw8ls50hlvqjic-groff-1.22.2

Does anyone know where I can see the error that Hydra is running into?

[vcunat]

I think this cached error is very old. It seems it would be best to restart the job.

[domenkozar]

http://hydra.nixos.org/build/4258635/nixlog/1/raw

[domenkozar]

Restarted, see http://hydra.nixos.org/build/5257601

[peti]

The groff issue is fixed in a3cc980.

[domenkozar]

@peti could you also cherry-pick that to x-updates branch?

[7c6f434c]

I finally got around to testing this on my Asus eeePC, which has an Intel 915 graphics card. (All my other systems have NVIDIA.) Everything (meaning Xfce) seems to work fine. I'm +1 on merging this (together with the stdenv-fixes branch).

StumpWM and Torcs work fine here on Intel video card, too

[tsuraan]

I'd like to try testing out the new x-updates branch (on my ancient Intel gfx laptop), but I can't find any docs for how to use the branch. I'm a total Nix newbie; I gather that I need to run nix-channel --add <something>, but what is that something?

[vcunat]

@tsuraan: I don't think there's a channel, you would probably have to check out the git versions of nixpkgs and nixos repositories (branch x-updates). However, I hope we merge it very soon to master (like this weekend?).

[tsuraan]

Ok, so once that's done I can just do a nix-channel --update and I'll be able to try things out?

[vcunat]

@tsuraan: yes, after some times (usually at most a few days) the changes to master propagate to the channel.

[peti]

Subscribing to

nix-channel --add http://hydra.nixos.org/jobset/nixpkgs/xorg-test/channel/latest

should do the trick. You may want to remove the original nixpkgs channel, though, to avoid mixing up those two package sets.

[vcunat]

True, I forgot hydra provides a whole channel for each jobset.

[vcunat]

It's all in master now, you can reopen in case of problems.