WIP: Make Paraview work by certik · Pull Request #3298 · spack/spack

certik · 2017-03-02T06:16:41Z

I use a stock Ubuntu 16.04. Paraview fails to build due to a missing opengl library. In Hashdist I build Paraview with os mesa, then things work. The +osmesa flag to paraview fails (missing osmesa library).

So I first concentrate on the mesa library. Below I made the modifications to build it just like in Hashdist (as the default spack version fails to build due to configure: error: libudev-dev or sysfs required for building DRI). I build it with spack install mesa@10.2.4 %gcc@5.4.0.

It fails with:

==> 'make' '-j32' 'autoreconf -fi'
make: *** No rule to make target 'autoreconf -fi'.  Stop.

But I have to call autoreconf, otherwise (if I comment the line that calls autoreconf), i.e.:

--- a/var/spack/repos/builtin/packages/mesa/package.py
+++ b/var/spack/repos/builtin/packages/mesa/package.py
@@ -65,7 +65,7 @@ class Mesa(AutotoolsPackage):
 
 
     def configure_args(self):
-        make('autoreconf -fi')
+        #make('autoreconf -fi')
         return ['--enable-osmesa', '--disable-dri', '--disable-egl',
                 '--enable-xlib-glx', '--disable-gallium-llvm',
                 '--with-dri-drivers=', '--with-gallium-drivers=']

the configure phase succeeds, but the make phase fails with:

==> 'make' '-j32'
CDPATH="${ZSH_VERSION+.}:" && cd /home/certik/repos/spack/var/spack/stage/mesa-10.2.4-gqowes7nvl5cnpgvpakilmzrcktjvnhs/Mesa-10.2.4 && /bin/bash /tmp/certik/spack-stage/spack-stage-_yG_LZ/Mesa-10.2.4/bin/missing aclocal-1.14 -I m4
/tmp/certik/spack-stage/spack-stage-_yG_LZ/Mesa-10.2.4/bin/missing: line 81: aclocal-1.14: command not found
WARNING: 'aclocal-1.14' is missing on your system.
         You should only need it if you modified 'acinclude.m4' or
         'configure.ac' or m4 files included by 'configure.ac'.
         The 'aclocal' program is part of the GNU Automake package:
         <http://www.gnu.org/software/automake>
         It also requires GNU Autoconf, GNU m4 and Perl in order to run:
         <http://www.gnu.org/software/autoconf>
         <http://www.gnu.org/software/m4/>
         <http://www.perl.org/>
Makefile:557: recipe for target '/home/certik/repos/spack/var/spack/stage/mesa-10.2.4-gqowes7nvl5cnpgvpakilmzrcktjvnhs/Mesa-10.2.4/aclocal.m4' failed
make: *** [/home/certik/repos/spack/var/spack/stage/mesa-10.2.4-gqowes7nvl5cnpgvpakilmzrcktjvnhs/Mesa-10.2.4/aclocal.m4] Error 127

I don't know what else to do. I tried to also debug it by doing:

$ spack cd mesa@10.2.4 %gcc@5.4.0
~/.../mesa-10.2.4-gqowes7nvl5cnpgvpakilmzrcktjvnhs/Mesa-10.2.4$ spack env mesa@10.2.4 %gcc@5.4.0
Traceback (most recent call last):
  File "/home/certik/repos/spack/bin/spack", line 212, in <module>
    main(sys.argv)
  File "/home/certik/repos/spack/bin/spack", line 208, in main
    _main(args, unknown)
  File "/home/certik/repos/spack/bin/spack", line 174, in _main
    return_val = command(parser, args)
  File "/home/certik/repos/spack/lib/spack/spack/cmd/env.py", line 62, in env
    build_env.setup_package(spec.package)
  File "/home/certik/repos/spack/lib/spack/spack/build_environment.py", line 513, in setup_package
    dpkg.setup_dependent_environment(spack_env, run_env, spec)
  File "/home/certik/repos/spack/var/spack/repos/builtin/packages/python/package.py", line 358, in setup_dependent_environment
    raise RuntimeError('Cannot locate python executable')
RuntimeError: Cannot locate python executable

So that seems like another, unrelated bug, so I reported it as #3319. But it prevents me from debugging it that way.

spack install mesa@10.2.4 %gcc@5.4.0

davydden · 2017-03-02T06:25:06Z

    homepage = "http://www.mesa3d.org"
-    url      = "http://ftp.iij.ad.jp/pub/X11/x.org/pub/mesa/12.0.3/mesa-12.0.3.tar.gz"
+    #url      = "http://ftp.iij.ad.jp/pub/X11/x.org/pub/mesa/12.0.3/mesa-12.0.3.tar.gz"
+    url      = "ftp://ftp.freedesktop.org/pub/mesa/older-versions/10.x/10.2.4/MesaLib-10.2.4.tar.bz2"


why do you change url, is 10.2.4 not available from iij.ad.jp? If so, just add it to the version() and keep the old url.

Yes, I wasn't sure if version() accepts a url or not, I was going to look into this later after I actually get it working.

it does, from http://spack.readthedocs.io/en/latest/packaging_guide.html#versions-and-fetching

version('8.2.1', '4136d7b4c04df68b686570afa26988ac', url='http://example.com/foo-8.2.1-special-version.tar.gz')

p.s. i checked the current url and the old versions can be found in ftp://ftp.iij.ad.jp/pub/X11/x.org/pub/mesa/older-versions/10.x/10.2.4/MesaLib-10.2.4.tar.bz2. So i would stick with this site.

Thanks, I fixed it in ee0f3a2.

@certik would you mind switching to the one i found above on the iij.ad.jp?

Sure, fixed in 9e30909. I already submitted the patch when you commented.

davydden · 2017-03-02T06:28:36Z


+
+    def configure_args(self):
+        make('autoreconf -fi')


i think configure_args is intended only to return flags, not to run anything in AutotoolsPackage package. Maybe @alalazo can comment on where this line should go.

It looks like the newer version fixes this autoreconf problem (it looks like spack actually somehow calls autoreconf automatically). I fixed this issue in a61d8ce, but I hit a problem that now I need to comment out the older version otherwise the url somehow fails to download it. So the way we implemented the url in version() above I think is incorrect.

Anyway, for now I am commenting out, and continuing on the quest to build paraview.

(it looks like spack actually somehow calls autoreconf automatically).

yes, AutotoolsPackage does it.

It looks like the newer version fixes this autoreconf problem (it looks like spack actually somehow calls autoreconf automatically)

Since #1186 sub-classes of PackageBase can declare an arbitrary number of phases to conform to the underlying build-system (cmake, autotools, etc.). The AutotoolsPackage class has logic in it to perform the most common operations (like call autoreconf if no configure file is present) so that packagers don't need to repeat them over and over.

spack install mesa %gcc@5.4.0

Build using: spack install -j16 paraview %gcc@5.4.0

certik · 2017-03-02T07:57:01Z

So

spack install -j16 paraview %gcc@5.4.0

now builds paraview on my Ubuntu 16.04 system. This is without GUI, but it's a progress. Now I am trying +qt.

certik · 2017-03-02T08:21:40Z

Ok, so this also built:

spack install -j16 paraview +qt %gcc@5.4.0

davydden · 2017-03-02T08:24:32Z


+
+    def configure_args(self):
+        #make('autoreconf -fi')


you can remove it if it's not needed. Probably same for the old version above.

davydden · 2017-03-02T08:25:20Z

    depends_on('qt@:4', when='+qt')

    depends_on('cmake', type='build')
+    depends_on('mesa')


should this be when='~qt', in that it's an alternative to qt?

it's not an alternative to qt, but rather to opengl.

i was concerned if we need that dependency unconditionally?

We might need a virtual dependency for opengl, and a set of recommendations for setting this stuff up. This was an indirect dependency already via qt; curious why @certik had to add it to paraview.

I do not think we should not have it unconditional:
As far as I know, If your system has GPU, you should use "native" Opengl and not Mesa
Is not there already an osmesa variant in ParaView?
Also did you compile Qt in spack or use the system provided?
I have the need to install Paraview + Qt on two cluster, one with GPU and another without.
On the one with I think that OpenGL is ( and has to be ) system installed, while on the one without, I nedd to install Mesa, and installed one with Gallium, as per
https://github.com/RemoteConnectionManager/spack/tree/wip/mesa_gallium
It is not ready but maybe worthwhile a WIP PR

This is just to get it working for me (thus the WIP in PRs name) in the most hackish way, it's another question how to best incorporate it or what the right solution actually is. I don't know.

@tgamblin I build Qt via Spack. I first build Paraview without Qt, so there was no dependency and the build failed. Let me try to build Paraview with qt and without this PR, it might actually work.

davydden · 2017-03-02T08:25:57Z

flake8:

var/spack/repos/builtin/packages/mesa/package.py:36: [E265] block comment should start with '# '
var/spack/repos/builtin/packages/mesa/package.py:67: [E303] too many blank lines (2)
var/spack/repos/builtin/packages/mesa/package.py:68: [E265] block comment should start with '# '

certik · 2017-03-02T08:28:18Z

So paraview built and I tested it a bit, it seems to work.

The mesa is more of a workaround, ideally one would build Paraview with opengl. Does anyone know what packages to install in Ubuntu to make this work?

For now I can use mesa, which is slower, but at least it works.

davydden · 2017-03-02T08:31:24Z

Does anyone know what packages to install in Ubuntu to make this work

ideally you would not need any, everything should come from Spack (but you can make Spack use system-provided packages as well!). Maybe some dependencies are not declared as such and thus opengl build fails in some cases.

tgamblin · 2017-03-02T08:58:01Z

@certik:

The mesa is more of a workaround, ideally one would build Paraview with opengl. Does anyone know what packages to install in Ubuntu to make this work?

Isn't Mesa the OpenGL implementation used in Ubuntu? Maybe we just need a faster mesa build.

certik · 2017-03-02T16:51:17Z

Btw, bad news --- Paraview worked over ssh yesterday, but now when I do it from the computer itself, it fails with:

$ /home/certik/repos/spack/opt/spack/linux-ubuntu16-x86_64/gcc-5.4.0/paraview-5.0.0-i5jo4cmyofjndfofftoworcrzop7jc2s/bin/paraview
X Error: BadDrawable (invalid Pixmap or Window parameter) 9
  Major opcode: 62 (X_CopyArea)
  Resource id:  0x0

So I still need to investigate.

certik · 2017-03-02T20:30:19Z

The previous error was fixed by unset GNOME_DESKTOP_SESSION_ID, no idea why.

Paraview seems to work, but the fonts in Qt look very crappy.

certik · 2017-03-02T21:20:21Z

Here is how it looks like:

The Qt fonts look really bad. Any ideas?

adamjstewart · 2017-03-02T22:03:27Z

@hartzell has messed around with Spack's font libraries before. He may be able to help.

certik · 2017-03-02T22:06:05Z

I noticed if I run qtconfig (I have to module load qt-4.8.6-gcc-5.4.0-3n374jg first), it allows me to change the default font in Qt (and it reflects in Paraview). Unfortunately, the selection is very limited and all the fonts look very crappy.

@hartzell if you knew how to increase the font selection available for Qt in Spack, that would be awesome.

certik · 2017-03-02T22:51:15Z

I think I figured out the font issue. I needed to install qt with webkit enabled:

spack install -j16 paraview +qt ^qt +webkit %gcc@5.4.0

It installs bunch of font libraries like fontconfig, I don't know if it's related, but then qtconfig has very good selection of fonts, and as you can see below, it looks good now:

certik · 2017-03-02T23:07:26Z

Here is a summary

How to get ParaView working on Ubuntu 16.04

Install spack and shell support. No patches necessary. Install prerequisites using apt:

apt install environment-modules freeglut3-dev libglu1-mesa-dev libgl1-mesa-dev libgles2-mesa-dev libgles1-mesa-dev libegl1-mesa-dev libxt-dev libxext-dev libdrm-dev

Maybe not all of them are necessary, but most of them are, otherwise Paraview and Qt will not compile properly. In case I forgot some library, here is a full list that I installed today: https://gist.github.com/certik/a02e697c38efdfba363b70bc918b2c82

Then:

spack install -j16 paraview +qt ^qt +webkit %gcc@5.4.0
spack load paraview +qt ^qt +webkit %gcc@5.4.0
unset GNOME_DESKTOP_SESSION_ID
paraview

certik added 2 commits March 1, 2017 22:28

Build with

bc57d8e

spack install mesa@10.2.4 %gcc@5.4.0

autoreconf is needed

cd4d099

davydden reviewed Mar 2, 2017

View reviewed changes

certik added 4 commits March 1, 2017 23:40

Move the url to version()

ee0f3a2

Use iij.ad.jp url

9e30909

Make it build

a61d8ce

spack install mesa %gcc@5.4.0

Make paraview depend on mesa

e88d8f5

Build using: spack install -j16 paraview %gcc@5.4.0

davydden reviewed Mar 2, 2017

View reviewed changes

certik closed this Mar 2, 2017

Conversation

certik commented Mar 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

certik Mar 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

certik commented Mar 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

certik commented Mar 2, 2017

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

certik Mar 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

davydden commented Mar 2, 2017

Uh oh!

certik commented Mar 2, 2017

Uh oh!

davydden commented Mar 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tgamblin commented Mar 2, 2017

Uh oh!

certik commented Mar 2, 2017

Uh oh!

certik commented Mar 2, 2017

Uh oh!

certik commented Mar 2, 2017

Uh oh!

adamjstewart commented Mar 2, 2017

Uh oh!

certik commented Mar 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

certik commented Mar 2, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

certik commented Mar 2, 2017

How to get ParaView working on Ubuntu 16.04

Uh oh!

Reviewers

Assignees

Labels

certik commented Mar 2, 2017 •

edited

Loading

certik Mar 2, 2017 •

edited

Loading

certik commented Mar 2, 2017 •

edited

Loading

certik Mar 2, 2017 •

edited

Loading

davydden commented Mar 2, 2017 •

edited

Loading

certik commented Mar 2, 2017 •

edited

Loading

certik commented Mar 2, 2017 •

edited

Loading