Feedback

[pzhristov]

From @cholmcc I really dislike alidist. It's bloated as all .... I prefer if I can install a distribution package, since I know that will work with the rest of the system. For those software packages for which there is no distribution package, I will often go to some length to see if I can make the package myself. If that becomes to cumbersome, then I will pull down the package myself and built it. That means I will often have master of a package for which ALICE uses some ancient version in alidist. Therefore, I sometimes see inconsistencies between ALICE software and third-party software that you would only discover the hard way through alidist. So there - because I can 😄

[pzhristov]

It is very simple - take alibuild and alidist as development packages and update them regularly, then you will have always the consistent set of software. And of course you can have many packages in development mode.

[pzhristov]

From @cholmcc alidst isn't as easy to deal with as you might think, and I'm not alone in that sentiment, and because I know how to things differently, I do that and get rid of the bloat of alidist.

[pzhristov]

From @sawenzel I think a constructive way forward would be to come to the relevant meetings (computing board, WP3 meeting) where the software packaging approach is discussed and present your (and the anonymous+) worries. This would be way better than ranting about them on the Github platform as side-remarks. In the end, choices about these systems were taken by our organisation some time ago and most people do not have problems following these common workflows (even in disagreement) because standardized approaches are a core foundation of working together efficiently.

[cholmcc]

This would be way better than ranting about them on the Github platform as side-remarks.

You literally asked me

ALICE is using a well-defined FairRoot version. What is the motivation to use something different than the one we compile against in alidist?

and so I answered (reprodued as the first post here). Then Peter chimed in with an assertive statement (reproduced as first comment here), to which I answered that I disagreed (reproduced as second comment here). Apparently Peter found that reply - to his own question - so egregious - that alidist isn't all that great - that it had to be deleted.

So if you believe that its bad to "ranting about them on the Github platform as side-remarks" then you must acknowledge that you were the one that asked the question and thereby started the discussion. I literally replied why I dislike alidist and my reasons for it - no one needed to do anything about that. I'm not advocating a single solution - rather the opposite in fact - at least for people who are willing to do that work. And that is really all I said - so not a "rant" in any sense of the word.

Yours,

Christian

[ktf]

A few pointers which might solve some of the issues you have. Notice how in general the documentation is at http://alisw.github.io/alibuild.

1. There is an aliBuild clean command, as documented at http://alisw.github.io/alibuild/user.html#cleaning-up-the-build-area-new-in-110. This will clean up the old packages which are not needed anymore.
2. There is plenty of hooks to reuse some well known system packages, the mechanism is detailed http://alisw.github.io/alibuild/user.html#controlling-which-system-packages-are-picked-up and http://alisw.github.io/alibuild/reference.html#the-header. In practice if there is a provide_system_check and the provided command passes, the system package will be used. You can check with "aliBuild doctor" to see why they are not picked up.
3. For some architectures we have precompiled packages, however those mostly require an updated alidist and a supported architecture.
4. There is support for "developer packages", i.e. you can checkout one of the packages, call it as specified in a recipe, and then aliBuild will pick up your version, rather than the one from alidist. This is useful, for example if you are adding a feature to ROOT and you want to have it picked up in an O2 build. The developer packages can actually be even built by hand, after the initial build with alibuild, and most of the work is in general done with cd sw/BUILD/O2-latest/latest && ninja install (or cd sw/BUILD/ROOT-latest/latest && ninja install for the ROOT example). If you are on a mac or you have direnv, you can even just run from there, no need for modules, moreover if your IDE supports direnv you can have the full development environment consistently setup.

What system are you using?

[cholmcc]

Hi all,

Let me expand upon why I dislike alidist (and alienv).

• First of all, I'm typically working on a laptop without the luxury of a large disk that can store a lot of software.
• Secondly, even though alidist is supposed to be able to use system packages it is far more likely that it cannot. This means, that you will end up with several version of the same library, which other libraries then use or may link to, and then those libraries need to be brought to sync with alidist and so the snow-ball is rolling
• Thirdly, if you, like me, are working on many different projects in many different domains, then you most likely would like to have only one version of a common package used by the various projects or domains. Otherwise, you may end up with conflicting versions, and even more software installations to maintain. If you're only working in one domain (say ALICE), then it can be made to work, but try adding just a small domain and you get a lot of trouble.
• Fourthly, the dependency tracking of module (the underlying utility used by aliEnv) is horrendous. You can easily import conflicting libraries and packages into a session and module will not object. That can cause all sorts of conflicts where one package requires one version of another while a third requires another version, conflicting, of the same second package. module simply doesn't deal with this at all.
• Fifthly, alidist tend to use ALICE-patched version of software, and if some other project uses that software, it may be led to believe that the software is what it wants, though it may not be the case at all.

There are actually ways of dealing with most of the above, but it requires a lot of discipline and some hacking. For example, an ALICE pathed library should have a distinct so-name so that it will not be loaded as a substitute for the real thing. Of course, that does not prevent an application to load both sets of libraries with possible conflicts arising. Another, and far better solution, would be to package up the software for distros. A third solution would be to run in docker, which then may cause some issues with development package.

So for me, the most effective way of setting up an environment, is to roll-my-own. What I typically do is have something like

/opt/sw/inst <- installed packages 
/opt/sw/build <- where packages are built 
/opt/sw/src <- source code of the packages

and where /opt/sw/inst is a regular FHS tree (bin, lib, include, share). The main hurdle in this case is to figure out for which packages I need an ALICE-patched version and for which I may be able to use the distro version.

Guilio asks which system I use: Debian GNU/Linux "sid" unstable - and there's another thing with using system packages - alidist tends to be centered on CentOS, and so on Debian it ends up pulling in a lot of stuff that it really doesn't need to pull in.

As for IDE's - there's one for any one serious about it: Emacs 😄 I case you cannot tell - I'm joking.

Wrt to "development" packages - Guilio says, you mainly build them yourself. In that sense, you can say what I'm doing is treating all packages "development" packages 😄 And in some sense, that is true. I will often make patches (and submit them upstream) to third-party packages - viz, this patch for Apache arrow.

Federico Carminatti initially made a very sensible decision way-back-when - to base everything on ROOT. It served as the abstraction layer that made it much easier to ensure consistency across platforms and software packages. It wasn't without its own problems, but you always had a well-defined environment - ROOT - to build on. ALICE now uses a plethora of platforms, and layers, which makes the whole thing a lot more complex: 1-to-N relations scale, M-to-N quickly becomes chaos.

My 2¢

Yours,

Christian

[ktf]

First of all, I'm typically working on a laptop without the luxury of a large disk that can store a lot of software.

Again, I suspect you are lacking doing some aliBuild clean. A typical workarea from O2 / O2Physics should not be more than 20GB, mine is 99GB (but I am clearly not the average user, yet I always develop on a laptop as well). Can you provide du -sh .../sw/* or do some investigations on what are the biggest offenders of your installation?

Secondly, even though alidist is supposed to be able to use system packages it is far more likely that it cannot.

aliBuild doctor would help here. While Debian is as supported on a best effort basis, like Ubuntu and macOS, it should work quite nicely. I suspect the problem comes from the fact you missed some dependency when adapting/ https://alice-doc.github.io/alice-analysis-tutorial/building/prereq-ubuntu.html.

Thirdly, if you, like me, are working on many different projects in many different domains, then you most likely would like to have only one version of a common package used by the various projects or domains.

Sorry, this I do not buy. aliBuild is all about being able to have several installations running at the same time, reusing as much as possible of what can be reused. There is options to support multiple alidist (-c), different build modes (--defaults ...) and even for branches (-z, which is there despite I found it myself an unneeded complication, when it was introduced). All of them support having shared build products. Could you provide an example of what you are doing with "multiple domains" which does not reuse the reusable as you would expect? Also notice that the direnv support makes switching different build environments immediate (and does not require modules).

Fourthly, the dependency tracking of module (the underlying utility used by aliEnv) is horrendous. You can easily import conflicting libraries and packages into a session and module will not object.

Agreed, but this is outside our control. The trick is to make sure that you use modules only via top-level packages (like O2Physics, or O2PDPSuite) and not try to load stuff one by one. Anything else, with or without modules, is deemed to fail (including what you propose with the system packages, BTW) unless there is a strict policy in what goes into the release. We had at some point some discussions about "streams" (kinda like Debian releases) but it went nowhere. Maybe we should resurrect the discussion.

For example, an ALICE pathed library should have a distinct so-name so that it will not be loaded as a substitute for the real thing. Of course, that does not prevent an application to load both sets of libraries with possible conflicts arising.

You answered to yourself about why that is a bad idea. I have seen that fail miserably 20 years ago. Having properly so-named stuff only works for very specific libraries and it's a mess to maintain.

Another, and far better solution, would be to package up the software for distros.

What you propose would result in only the production architectures being realistically supported (i.e. no debian), and it will never be accepted by the Grid sites (which support 100% CVMFS). Moreover, it would prevent multiple installations, which are needed again on the grid. The only place where we do what you propose (repackaging alibuild tarballs into system level RPMS) is the FLP cluster where one of the top complaints is the lack of parallel installations and quick rollbacks.
Anyway, if you want to go that route for your own installations, you can probably try to use the "template" option of alibuild build with a custom template which works for Debian to produce debs. Such mode allows you to get all the packages, their recipes and metadata in a jinja template. I can provide you a gist as a starting point, if interested.

A third solution would be to run in docker, which then may cause some issues with development package

Simply use the --docker option, rather than doing things by hand. It handles development packages as well.

As for IDE's - there's one for any one serious about it: Emacs 😄 I case you cannot tell - I'm joking.

That explains many things... :-D Have a look at the direnv integration for emacs, e.g. https://github.com/wbolster/emacs-direnv. I have never tried (I use vim's one ;-)) but it should work just as well.

Hopefully the above answers to at least some of your problems.

Federico Carminatti initially made a very sensible decision way-back-when - to base everything on ROOT

https://youtu.be/lXKrRBYhxeA?t=197

[cholmcc]

Again, I suspect you are lacking doing some aliBuild clean

Since I'm not using alidist (or alibuild), the point of clean is kinda moot 😄

My directory with the software (/opt/sw) is some 150GB. And I've cleaned several builds (Geant4 for example).

aliBuild doctor would help here.

Actually, it didn't when I tried. It got so many things wrong that it was evident that it was not the right way for me to go.

I suspect the problem comes from the fact you missed some dependency when adapting/ https://alice-doc.github.io/alice-analysis-tutorial/building/prereq-ubuntu.html.

First off, how do you know it is a fact? (Maybe it's a language thing 😄) No, I didn't get the dependencies wrong. I've been using Debian since Debian 1.3, so I know my way around Debian. I'd prefer if you didn't make assumptions if this nature.

Sorry, this I do not buy. aliBuild is all about being able to have several installations running at the same time, reusing as much as possible of what can be reused. There is options to support multiple alidist (-c),

You are talking within the ALICE world. As soon as you add other worlds - or as I termed them domains - into the mix, then these points go moot.

Agreed, but this is outside our control.

Not really. You could have chosen a different "packaging" system.

Anything else, with or without modules, is deemed to fail (including what you propose with the system packages, BTW) unless there is a strict policy in what goes into the release.

Proper dependency and conflict tracking (the latter is sorely missing from any RPM platform) is the entire point of a packaging system, so if all ALICE software was provided as system packages (DEB, RPM, ...) then you would use the regular system tools (which generally are far better than most other solutions - even rpm 😄) to ensure that there are not conflicts and all have their proper requirements.

You answered to yourself about why that is a bad idea. I have seen that fail miserably 20 years ago. Having properly so-named stuff only works for very specific libraries and it's a mess to maintain.

Actually, proper so-names is at the foundation of a sane packaging system. It is the only way that at run-time one can detect conflicts and load proper libraries. On Debian, for example, you can easily have multiple version of the same library libfoo and different applications, bar and baz, may load these different libraries. This can only be done if the so-names are meaningful. Maintaining so-names was solved some 30+ years ago with libtool (but CMake has since then messed that up entirely - progress, you know).

Hopefully the above answers to at least some of your problems.

I'm afraid not. The problems still persist. Perhaps I wasn't clear enough that I did actually try very hard to use alidist but quickly found that it didn't really work for me. Also note, that I'm speaking for my own use, and I appreciate that there are other kinds of users. But also know that I do have considerable knowledge and experience with these kinds of things, so I'm not talking out of my ... - well, you get the idea 😄

https://youtu.be/lXKrRBYhxeA?t=197

I'm not sure I get the joke. Any case ...

Thanks.

Yours,

Christian

[cholmcc]

Also notice that the direnv support makes switching different build environments immediate (and does not require modules).

direnv does not seem like a "solution" - rather it seems like a giant complication. It reminds me of people putting all their environment settings into ~/.bashrc instead of ~/.profile. That's a really bad idea - sub-shells, and there are many of those, will override your environment. The whole point of the environment is that you can change at any point and you always know exactly how you changed it. Anything that "magically" changes the environment is a no-go.

My direnv is that I have /opt/sw/inst/etc/profile which I source when I need software from there. Its then localised to that shell, and I know what's going on. But this (and direnv) is a far cry from "package management".

Yours,

Christian

[ktf]

Since I'm not using alidist (or alibuild), the point of clean is kinda moot 😄

Sorry, I think I misunderstood where you were coming from. I thought you were using alibuild and things were not working for you, hence I tried to help you there.

You are saying instead that you are not using alibuild and things in the stack are too complicated for you to build with your own method. Fair enough, but keep in mind that is unsupported and any help you get is because we care about O2.

All I can say, is that part of the effort we do is always in simplifying dependencies and reduce the need for custom patches. Sometimes we manage, sometimes we don't, sometimes things are historical and it takes a long shutdown to be cleaned up (e.g. the dichotomy between ROOT5 and ROOT6 builds). aliBuild is what allows us to manage the complexity and it's the worst of all package managers (apart from all the others).

Personally I think you will be better off simply adopting alibuild and learning how to use it. The result will for sure better of what you currently have.

ALICE software was provided as system packages (DEB, RPM, ...)

We do provide RPMs for the production platform, but then you are stuck with CC7 (Alma9 now). Again, if you want to create deb files from a given alidist incarnation, I can tell you how it can probably be done with a plugin, but the effort is up to you.

proper so-names is at the foundation of a sane packaging system

Yes, too bad in our field proper so-names are not a thing and you have incompatible libraries which do not bump the minor version, you have the diamond dependency problem which would require using versioned symbols (which again works only for certain things), etc. etc. You can rant all you want about CMake (or Debian as a matter of fact), but that ship has left the harbour a long time ago and you need to deal with the fact that proper so-names are not a thing and that the moment you use at the same time two incarnations of the same library, you crash.

direnv does not seem like a "solution"

Ok, nevermind then.

I'm not sure I get the joke. Any case ...

Have Federico explain it to you.

[davidrohr]

Just to add few comments:

• A real problem I see with alidist is the tons of system packages, which we are regularly building. I regularly have collisions with libraries installed on the system due to that on my Laptop running Gentoo. And even on RHEL systems I had problems with the precompiled binaries if the installed system packages differ from what we are building for. What I am doing now:
  • On RHEL docker containers, I always disable the remote store. Takes ages to build, but at least it works, and I have plenty of cores...
  • On my Gentoo Linux, I have disabled all system libraries in alidist, and force using them from the system, including arrow, clang, root, but not the transport because building Genat etc. in the system ended in a mess. This means that I have to keep the libraries up to date in the system, which is ok for me. Only thing I cannot get to work at the moment is ONNXRuntime, since it requires an ancient version of flatbuffers, which I cannot even build any more with my new GCC... So for now I disabled ONNX fully.
• What I do not like in aliBuild are the defaults, particularly the fact that one default sets all settings, thus we need o2, o2-dataflow, o2-epn, .... A system like use flags in Gentoo would be much more complicated.
• For the size of the sw folder, I got myself an 8 TB NVME SSD as soon as they became available... Happy not to be stuck on a Mac :).

[cholmcc]

You are saying instead that you are not using alibuild and things in the stack are too complicated for you to build with your own method. Fair enough, but keep in mind that is unsupported

Which is why I don't expect support

and any help you get is because we care about O2.

but I do expect you to listen when I point out that certain things are broken in environments that may stress test the code, exactly because "we care about O2."

All I can say, is that part of the effort we do is always in simplifying dependencies and reduce the need for custom patches.

Vit the latest stuff on Rivet - there are certainly ways of avoid patches that are not explored in alidist, so the statement

it's the worst of all package managers (apart from all the others).

... is something of an overstatement. I would venture that PIP does a better job at it then alidist does. I do not have too much experience with npm but that seems to work reasonably well too.

Yes, too bad in our field proper so-names are not a thing

I do not understand what you mean by this. Libtool solved this issue a long time ago. Just too bad that CMake messed it up. In some sense, it is quite simple - bump the major version if you change the ABI.

and you have incompatible libraries which do not bump the minor version,

That's actually rare this days - except HEP software like ROOT or $O^2$. Debian has been really consistent in driving this home with developers. Now, not all follow it - vis rapidjson that changes their API (not ABI because it's headers only) without bumping version numbers.

BTW, if $O^2$ would use so-names, then a lot of conflicts could be avoided. Of course, most of $O^2$ has moved out of libraries and into applications - essentially moving the interface to be data - so the problem is smaller than in AliROOT, but it is still there.

you have the diamond dependency problem which would require using versioned symbols (which again works only for certain things),

Versioned symbols are really only something you see in deep system-level libraries like libc or libstdc++ and are meant to provide backward compatibility - i.e., you can have an application using an older ABI resolve its symbols against a library that by default uses a newer ABI. It is not really an issue for higher level libraries. Diamond dependencies are like

     A
   /   \
B       C
   \    /
     D

but here the identifiers A, B, C, and D are all versioned. For example

        A-1.0
        /     \
B-2.0      C-3.0
        \    /
      D-4.0

which is fine. But if C-3.0 depend on D-5.0, then you have

        A-1.0
        /      \
 B-2.0      C-3.0
       |         |
   D-4.0    D-5.0

which would not work, and would be caught. module does nothing to prevent this situation - i.e., you could have built A in the top environment, but then run it in the bottom environment (other than rely on the so-names to fix it for it), since module doesn't track dependencies very well. David is right in that if you stick to meta packages like o2 and never load anything else (i.e., stick to the "sanctioned"/"stifled" environments) then there should not be a problem.

Of course, things get even worse when code can dynamically load (dlopen - not linker) other code.

When I build things myself, I can better ensure that I don't get into these situations. Sure, I may need to fix up things here and there, but no more than I would need if I used alidist. Remember, this whole thing sprang out of Sandro asking why I do not use alidist, and I replied why I do not like it.

etc. etc. You can rant all you want about CMake (or Debian as a matter of fact), but that ship has left the harbour a long time ago

It's not a rant - look up what rant means. The problem is, as I see it, that we had a solution but then CMake came along, and with the lure of portability regressed on that solution. CMake might be big, but there are certainly many other viable solutions out there and some of them are right up there with CMake (if not bigger) in terms of adoption.

and you need to deal with the fact that proper so-names are not a thing and that the moment you use at the same time two incarnations of the same library, you crash.

First of all, it is not a fact (nor does it make it one that you stated like that) that so-names is not a thing - plenty of platforms would vehemently disagree with that. Secondly, you will not be able to load to versions of the same library if they use proper so-names - that's the whole point. I think, perhaps, you've missed something about so-names.

Have Federico explain it to you.

I'm not likely to run info Federico these days, so I was hoping you could explain it - my Italien is a bit rusty.

BTW, now I have your ear - could you check out this PR on $O^2\mathrm{Physics}$ - also the comments - thanks.

Yours,

Christian

[ktf]

Which is why I don't expect support

Good.

but I do expect you to listen when I point out that certain things are broken in environments that may stress test the code, exactly because "we care about O2."

I listen, thank you for your feedback.

That's actually rare this days - except HEP software like ROOT or O2.
...
things get even worse when code can dynamically load (dlopen - not linker
...
I think, perhaps, you've missed something about so-names.

I think you missed the fact that we are HEP, that all our software uses CMake, ROOT, rapidjson, old flatbuffers and all the packages which you mention as exceptions and that we do things like loading dynamic plugins (not to mention the ROOT dictionaries). When you manage to convince our dependencies like ROOT and GEANT4 to follow your proposed model, I will be happy to push for the required changes in the upper part of the stack. Until then, we have an experiment to run. I am afraid "things usually work" is not a viable strategy for us.

David is right in that if you stick to meta packages like o2 and never load anything else (i.e., stick to the "sanctioned"/"stifled" environments) then there should not be a problem.

I think it was me, actually. Yes, that's my recommendation, stick with the O2PDPSuite metapackage.

BTW, now I have your ear - could you check out this PR on $O^2\mathrm{Physics}$ - also the comments - thanks.

I can only second what a few others have already told you. Come with a proposal to WP4+WP14 and we discuss there.

[cholmcc]

I think you missed the fact that we are HEP, that all our software uses CMake, ROOT, rapidjson, old flatbuffers and all the packages which you mention as exceptions and that we do things like loading dynamic plugins (not to mention the ROOT dictionaries).

We're discussion how thing could be done differently - not what need to be done differently.

Not all software uses CMake - work from the bottom up, and I think you will find a lot of code uses something different. It is certainly not true that all upper-level code uses ROOT or is interfaced through ROOT. If that was the case, then we'd be back to how things were in 2000.

Side note: fact is rarely the appropriate word when making a statement. It presumes evidence that often isn't forthcoming. It does seem to have snug its way into AliNglish though 😄

When you manage to convince our dependencies like ROOT and GEANT4 to follow your proposed model,

Geant4 is not so bad - but ROOT definitely is 😄 - I should know - I'm a long time contributor to ROOT 😄

I will be happy to push for the required changes in the upper part of the stack. Until then, we have an experiment to run. I am afraid "things usually work" is not a viable strategy for us.

I would say "things usually work [@ CERN]" is the approach of alidist - at least in my experience - and I don't think I argued for that approach in any way. In fact, I think I argued quite the contrary that robust software development requires testing as many corner-cases as possible - quite the opposite of "things usually work".

I can only second what a few others have already told you. Come with a proposal to WP4+WP14 and we discuss there.

messages must have crossed paths.

Now I have your ear: Please have a look at this message. Thanks.

Oh, and if you have some input on this or this I'd love to hear

[ktf]

@davidrohr @pzhristov @cholmcc maybe we should take the whole discussion to an ALICE forum...

A real problem I see with alidist is the tons of system packages, which we are regularly building.

What you call "system packages" on the Grid were not present in a consistent manner, hence the need to bring our own stuff for that usecase. Nothing prevents us from using system python or openssl in general, however until very recent (in HEP terms) we had python 2.7 or openssl 0.9.8 as system baseline for our production platform. It not like we built a patched OpenSSL because of the fun of it...

That said, the --disable flag was introduced exactly to do what you do: pick up system tools where possible, as you do. We should probably start to exploit the new mode @TimoWilken introduced to properly track system software.

That said, to me the issue is not system software, apart from maybe clang, all the rest builds in a blip. The real time consuming parts are ROOT, GEANT4, the generators, and our own stack, which we will never be able to consistently distribute on all the platforms we effectively support.

Only thing I cannot get to work at the moment is ONNXRuntime, since it requires an ancient version of flatbuffers, which I cannot even build any more with my new GCC... So for now I disabled ONNX fully.

Which is actually an issue with ONNX itself, which Microsoft does not seem to want to tackle. Hence the need for an obsolete flatbuffer in our stack.

Again, I sympathise with the problem, but you know better than me that we have some very strict boundary condition at time...

Anyway, since this is becoming a bit too "ALICE strategy" specific, I suggest we move the discussion to an ALICE forum.

What I do not like in aliBuild are the defaults, particularly the fact that one default sets all settings, thus we need o2, o2-dataflow, o2-epn, .... A system like use flags in Gentoo would be much more complicated.

Again, my answer to this would be "ALICE strategy" wrt CVMFS and not for this forum.

For the size of the sw folder, I got myself an 8 TB NVME SSD as soon as they became available... Happy not to be stuck on a Mac :).

I have half of my 2TB empty... ;-)