An experience with NixOS. #4952
Comments
|
I tried building SBCL-1.2.5, cutting out the SB-POSIX tests until I can figure out something better. I'm betting that this is NixOS related. there were issues trying to run an XULrunner binary too for whatever it's worth. |
|
Some notes:
It's been known for a while that unfortunately documentation is lacking on a few of these fronts. It's absolutely something we need to fix (I too think the NixOS on-boarding and beginner material is fairly sub-par).
In your paste of the Emacs That said, I'm afraid I cannot help you with SBCL or StumpWM. I use neither. As it stands today, the NixOS community is structured such that all Nix users are, in essence, de-facto Nix developers. This community is still so small it is almost a requirement for you to use it regularly. In return, most of us believe the benefits are massive enough to deal with that. But it is not true of everyone. That is an understandably painful reality, but it's a reality we cannot yet change. Hopefully in the future we can have enough users and dedicated maintainers to alleviate this burden from the userbase, and StumpWM and SBCL and X number of other things will just work for anyone who wants them, with no effort. If a package is broken or outdated or unmaintained (which happens with any distro), it almost always falls to a user who wishes to use that software to maintain it, I'm afraid. Have you tried asking on the Oh, and one final thing about your 'editors note' (which I'll quote to ensure everyone can read it for the future): If you wish to behave this way, I'm afraid my impression is that your mind about how to conduct yourself in public spaces is fairly made up - and no amount of honest actors will change that behavior or attitude. I can understand frustration at using new software and being perplexed by it. But your tone in here is pretty clear. And I imagine those aren't attitudes or tone we want anyway - besides, nothing brought you here beyond your own desires. So, if that is the case, maybe it would be best for you to find a community that does appreciate that sort of attitude - perhaps the Common Lisp community does. Or the Xbox Live community, perhaps. |
For worse. I would use the term "tedious bullshit" to describe this task, but whatever, moving on. I'm an idiot, thank you. I'll look into this now. WIP, K. curl http://nixos.org/irc/logs/log.20141112 | grep gabriel_laddel (dolist (k '("define remove" "define fix" "define test") (google k))) There is nothing perplexing about Nix. A person with the handle thoughtpolice doesn't want anyone to use mean words or to be critical of a project he is involved with? Color me shocked. ... sizzle Thanks for the help. |
No, actually. It's because I'm a developer who works on other open source projects (several for my job), and realistically I have better things to do than deal with annoying people telling me and other people in the projects I work on to go fuck themselves - which substantially decreases my motivation to deal with said project in the first place. Indeed, my respect lies with the NixOS developers, and not you. So you'll have to forgive me if I find you annoying, but that's life sometimes. This should be pretty plainly obvious (as opposed to your quite rude assumption of my motivations and presumably willful misinterpretation of it, as if it would help your position). But apparently it needs to be spelled out to some people who are seemingly incapable of understanding these aspects of social interaction and why their behaviors might be considered harmful. Which I believe are pretty clearly alluded to in my original posting. Of course, I'm not about to succumb to such belittling tactics by people like you - that would only prove such tactics work to drive people away, ultimately harming everyone. So I'm afraid the feared Orwellian Thought Police of NixOS™ is here to stay -- much to your great despair, I imagine. Be afraid - be very afraid! That said, this bug is a bit convoluted. Really there are several things going on here, based on your original description:
For example, when we use GCC and specify a library like This explains why we have packages like Then, individual SBCL packages would go into their own namespace, such as
Really though, these are all separate issues in a way, so I would suggest you file them each in a ticket so they can be triaged and more easily tracked by developers. Having a meta-ticket is fine in general AFAIK (maybe 'overhaul SBCL support' which most of the issues would fall under), although keeping separate issues for the separate subtasks makes it easier to manage them and address individual concerns. Of course, I'm not going to do this for you, but I'd suggest that it's probably the easiest way to at least have your complaints looked at (as opposed to posting a very long Emacs buffer - the syntax highlighting actually makes it rather painful to read).
Don't worry about it - although you won't be getting any from me in the future. Although I will certainly return and be sure to call you out on shitty behavior should I see it. After all, what kind of Thought Police™ would I be if I didn't? I'd be letting you down, what with the expectations you set for me! |
|
This issue is fascinating, I've never seen a bug report written in LISP before. I'm also a little perplexed by the general tone and sense of entitlement but anyway... I'd like to address some of your points @gabriel-laddel . I see that you try to build EMACS from the command line. That's not how it is done in NixOS as that's not repeatable and stateful. Either you clone nixpkgs and change the nix expressions already there, or you use the packageOverrides in your You asked 1 question on #nixos and then logged off when you got no replies? Perhaps the nix-dev mailing list would be more suited to your IRC style. I general I would say there is a passion for code quality and statelessness in NixOS, just go look at all the comments on issues asking to change this or that. You seem to have gotten the reverse impression from the learning curve and EMACS. Nothing I can do about that, just interesting. If you like the Nix concepts you could implement StumpWM the way you like it, or you could just walk away. I don't know what causes your SBCL error. The guys in the LISP channel are correct, when you build something for Nix you fix all inputs, and if something doesn't work like that you wrap it so it only sees the part of the world that you want. Oh, those sb-posix test failures are also perplexing. Expecting 13, getting /? Wow. Of course, just turning them off was not the right thing to do. Anyway. |
|
Oh and hats off to @thoughtpolice for his detailed and gracious answers. |
|
Oh, and I will leave one mention about the IRC channel: I am not sure in what timezone the OP resides, but realistically from my time contributing I've noticed NixOS has a very large European userbase and a much smaller "not European" userbase. Given the assumption the logs are timestamped to a European timezone, it's perhaps not surprising most people weren't available in the early morning hours. So knowing nothing about OP, I imagine this could possibly have some impact on the availability of people to respond. As an American myself, this too has caught me off guard. So yes, the |
|
Also, currently NixPkgs has a working StumpWM 0.9.8 package (previous release) compiled using SBCL-1.2.5 (which is also in NixPkgs). Some of SBCL tests make assumptions about the base system; the others are run during the SBCL build: http://hydra.nixos.org/build/16738664/log/raw (look for ACLREPL-TESTS) |
There's an SDF representation of the syntax in @edolstra's PHD thesis starting on page 64, though I'm not sure if that would be entirely up to date. It also can be seen as acting as a design document, and lays out the thought and central metaphors behind Nix/OS pretty clearly. |
|
@Shados actually, you could say that https://github.com/NixOS/nix/blob/b6809608cc467925db44b1eb435095c37e433255/src/libexpr/parser.y#L298-L523 (Bison configuration) is a very strict, machine-readable BNF form... I wonder if there's a tool that takes Bison input and generates a human-readable grammar. |
|
I've seen everyone's responses and will be responding in due time. |
|
Relevant, apparently you can get an almost-BNF out of bison with |
|
FWIW, here's the grammar. The tokens are defined in https://github.com/NixOS/nix/blob/master/src/libexpr/lexer.l#L80. |
|
There has been no activity for a couple of weeks, so I'll close the issue. If there are actionable items hidden in this thread somewhere, I guess it would be better to create separate, new tickets for those. |
|
I'll note that I've drafted my response to all unanswered messages but am busy and don't know when I'll have time to finish it off. Closing the issue is fine with me. |
|
@gabriel-laddel just paste your draft? Otherwise you'll never come back to it... |
|
I don't publish drafts. I'm in the middle of something at the moment and will respond over the weekend. |
|
:-) reminds me of classical Cathedral vs. Bazaar dilemma. |
|
Wasn't able to finish over the weekend and am busy for the next two days. Best guess is wed/thurs. |
|
Note1: I've read NixOS: A Purely Functional Linux Distribution and Nix: A Safe and Policy-Free System for Software Devlopment, but not the associated PHD thesis. The contents listing didn't indicate that it'd introduce me to any new ideas. Note2: I've ignored forth and apl derivatives in this comment. Anyone familiar with will understand why upon reading it. Note3: I posted and deleted this approximately ~5min ago - sorry, the using ``` as blockquote didn't work as expected. Fundamentally, package management is composed of graph traversals and transformations. The goal that separates NixOS from other package managers, reproducible builds, can be rendered as "the ability to save and restore unix's dependency graph". This is a neat idea, but the implementation merely trades one set of problems for another. Many crucial design decisions that should have been throughly discussed were completely ignored. For example, nowhere in the design documents is it stated that every(?) programming language already has its own package manager(s) with its own idiosyncrasies that for the most part don't share concepts or code with the others. There are many good questions that originate from this fact, were one to acknowledge it. Examples: at what point is appropriate to leave package management up to a language's ecosystem? Is there ever a reason to? Can maintainers be convinced that the 'nixos way' is a good one and that packaging their code with a flag of some sort that produces a nix friendly build script is a reasonable request? Are we confident enough in nix to spend their time? Given that the NixOS plan is to modify build scripts and sources to its liking, what is the proper way to go about doing source-to-source transformations? Are there commercial offerings that solve this problem? The undiscussion of these issues is prototypical of the Nix design documents and documentation. From NixOS: A Purely Functional Linux Distribution,
How exactly does the reference scanner detect runtime dependencies? Regex and sed hackery? Or the correct way, parsing each language into its abstract syntax tree (ast), walking that and finding references to unix level dependencies (both static and dynamic)? I assume the former, as the latter is a great deal of work and would have been mentioned. Also, the only complete C99 parser (i.e., one that takes into account all the GCC extensions used in the linux kernel) I'm aware of is the haskell package Language.C.AST - and nix doesn't use haskell. The problem with anything other than traversing a language's ast is that you're forever stuck with an approximation of the input language, resulting in false positives etc. This is not new information.
Notice that the paper says,
but fails to mention how many people used NixOS over those 5 years, how many times undeclared build time dependencies /outside/ of the Nix store were encountered or how many packages were in the Nix store over this time period. Is this not relevant information? As Naggum stated above: "the rub with these 90% solutions is that you have /no/ idea when they return the wrong value becuase the approximation destroys any ability to determine correctness." If say, most of the NixOS users are NixOS developers (which they are) and the reference scanner doesn't walk the ast of the programs in question (which it doesn't), no reported failures means exactly nothing, because there will be many code paths not encountered by its users due to the sheer complexity of the software being interfaced with (e.g., how many NixOS +developers+ users use emacs docview? Not many apparently). There are other issues with these papers I'm going to skip over, such as the clearing of environment variables (the proper thing to do imo is to gensym them to prevent unwanted interactions and study how they're being used - filter for 'functional' builds...), failure to discuss why a new project is needed in the first place (why can't you just perform this computation via an extension to an existing project?), total wtf issues with the papers (who are they written for? One who needs the concept of "file system" explained is going to have nfi what emacs lisp or ~/what/all/the/slashes/signify) etc. The heart of the matter is that unix (ostensibly[0]) plays host to many different philosophies of how to go about programming. Languages differ in their approach to build systems, package management, syntax and foreign function interfaces. To change the way many software packages are built in a semi-automated, non-ast-walking fashion is insane. Not to discuss this in the design documents is insane. NixOS has 46 contributors and the documentation is sufficiently general as to give each contributor his own wildly different interpretation of the scope of the project. As far as I can tell, were one to extrapolate from the given information to a set of concrete requirements we see that NixOS plans to rewrite the build scripts for every version of every project on unix. Again, this is insane. The correct thing to do in this situation is to realize the utter impossibility of the task that has been set forth and re-evaluate one's approach.[1] Let's examine the reference scanner. What is the correct way to approach the problem? We know regular expressions cannot respect the semantics of a language and thus are (unless used as a part of a larger parser) inappropriate for meta-programming (source-to-source transformations). Thus, we must work by manipulating the ast of various languages, and then pretty printing the altered code into the language's concrete syntax. Most languages don't offer the ability to do this, and if one examines e.g., Clang, he will see all sorts of nonsense about concrete/surface syntax, context free grammars and compiler technologies. This certainly can't all be necessary (hint: it isn't). Consider the following mathematical expression: (3 + 2) * 8 / 3 * 3^6 Fully parenthesizing yields: (((3 + 2) * 8) / (3 * 3^6)) When computers execute programs, or humans mathematics, the order of operations must be taken into account. one way to notate the previous expression to remove such ambiguities present in mathematical notation is to move operators to the front of each parenthesized list, passing the remaining elements as its arguments. This is known as fully-parenthesized prefix notation. (/ (* (+ 3 2) 8) (* 3 (^ 3 6))) This notational scheme has a direct mapping to the program's ast, which can be rendered as http://i.imgur.com/8TO5VcK.png See the pattern? An s-expression (fully parenthesized prefix notation) is merely a serialization for the ast. Thus, manipulations of the sexpr correspond to manipulations of the ast. Languages with the vocabulary to manipulate the s-expression structure can manipulate themselves (their ast) just as easily as any other thusly structured information. It follows from this that all software 'tooling' and translation tasks, are transparently tree traversals e.g., find me all things that call this function, reference this variable, modify all build scripts to take into account the new &optional argument in the updated library, etc.[2] For whatever reason, many find s-expressions distasteful and spend much of their lives attempting to add the same meta-programming facilities to ALGOL derived languages. They all have failed. Not because of any sort of mechanical failing of the computer, but the human's inability to fully comprehend the parsing and syntactical schemes they're able to create (Ruby's parser is 10k lines of C, Clang's is >100k loc. Note that the entirety of SBCL is ~300k loc, and has much fat that could be trimmed off - http://www.cliki.net/Lisp%20-%20Next%20Generation). Scala is a notable failure in this regard. Watch this video: http://www.youtube.com/watch?v=TS1lpKBMkgg Pay attention to 37:39-42:50 and you'll get to see Paul Phillips flipping out over ir/asts (same thing!). He even states his plan for the next 25 years - attempt to solve a problem solved 50+ years ago (http://c2.com/cgi/wiki?LispOnePointFive). In particular, I found these quotes quite pertinent. "I want to programmatically generate asts and feed those" (not everyone does this, just algol derivatives) "The ast is going to be designed along side the VM" Wait, like every common lisp compiler ever? 30+ years behind the times yo. "the code that you look at, that ought to be a reflection of the AST. The canonical thing ought to be the tree, the code is a view of it.... It's trees that are fundamental, that's what we work with" Lol. Gotcha. "something not offered by our tremendously entangled compiler, which doesn't even have a clean parse tree. It's comical. Try to get back to the source from what you get out of the scala parser. To me, the minimum test of a parser is that it parses!" Lol. 'Comical' is definitely the right word. "modifiability is paramount. If it isn't straightforward to modify, it will never be any good. It will never be fast. It will never be correct. And it will eventually be replaced by something modifiable... after consuming as many hours as you feed it." Again, 30+ yrs behind the times[3]: http://article.gmane.org/gmane.comp.java.clojure.user/34272 So what does all this have to do with NixOS? Much of the complexity in NixOS originates in the failure to address why it isn't a trivial modification on an existing project. Were they to have honestly sized up the problem they'd end up at something like: people suck at both reading & programming, also asts are fundamental. They'd then realize that at some point you have to choose some languages and leave others, because not everyone has something to offer and writing fully compliant parsers for all the languages involved is about as entertaining as building pyramids. When one has decided on a particular scheme, one could set about implementing "the ability to save and restore parts of unix's dependency graph". But by failing to address this complexity NixOS inherits it, and in practice, adds to it by the 'invention' of /yet another/ language created for no particular reason. 800+ existing languages[4] and /none of them/ have the required properties? Nonsense. Allow me to anticipate your retorts: "I need to know that my nix code is functional" "I don't want to require a whole compiler!" By failing to make use of existing programs NixOS not only adds a language (compiler/interpreter), but the requirement for an Emacs mode (+1 more for every other editor!), custom syntax highlighting, extra crap for auto-completion etc. etc. One should instead make use of what already exists. As it stands SLOCCOUNT reports 26.3k loc, divided across 7 languages being are used to introduce another for the nix project. This number of languages alone is enough to guarantee that no one will ever understand the whole codebase on a rolling basis and have enough time to do much else with their life. A preceptable analogy to the natural languages springs forth: if we created them for the same reason so-called professional programmers create programming languages, each new subculture would require a whole new alphabet/glyph scheme, phonetic system, grammatical structures etc. Though some surface features may be shared across languages, the underlying semantics they represent would change in unpredictable and often incompatible ways.[5] "So, you want us to write lisp? What if something better comes along?" You are faced with a choice - either learn from those who came before you, or spend the rest of your life fighting monstrosities of your own creation, attempting to eke out some order in all the chaos, be it the SDF/BNF format of a language's syntax or otherwise. In any case, if something better comes along, transforming your s-expression code into $POWERFUL_NEW_NOTATION will be a straightforwards intern-level task[6]. Now to address some unanswered questions.
You don't say.
You're in Mexico city and ask a bystander to direct you to the airport (your phone died, you lack a map and you're going to miss your plane unless you make it to the airport soon). He gives very specific instructions, which you write down and follow exactly. You reach the last instruction somewhere near the slums, missing your flight. As such, you swear loudly. Is anyone entitled to correct directions? Of course not. I however, expect the /bare minimum/ of people giving correct directions and accurate descriptions of their projects, much like I expect my friends to brush their teeth regularly.
If I say "jump in front of the bus on my mark, ready? 1, 2, 3 - JUMP!" it is clear what I'm asking/ordering, what is less clear is why I'd be asking/ordering someone to jump in front of a bus.
There are very few classics in the field of computing. We are at the very beginning of this journey and it hasn't even begun to get interesting yet. IMHO, the Cathedral vs. Bazaar isn't in anyway a dilemma or a classic. Linux was a failure 20 years ago, it is a failure today and any posturing otherwise is just that. See The UNIX-HATERS Handbook[0] for more information. I find the contrast between these quotes accurately depicts the computing situation circa 2015.
[0] In actuality unix is mostly a pile of stupid, see: web.mit.edu/~simsong/www/ugh.pdf [1]
[2] This scheme the surrounding ideas are so fundamental that they've sometime's referred to as "Maxwell's equations of Software". You can read more about Lisp and s-expressions elsewhere so I'll not repeat it here. http://www.defmacro.org/ramblings/lisp.html [3] This information re, Scala is interesting beyond the technicalities. Typesafe, the Scala company (has Martin Odersky as Chairman and Chief Scientist) has received 31 MM in funding. Coursera (who built everything in Scala) has received 85MM. I'll predict they'll eventually fail as they're out-competed by more intelligent adversaries, though both will (due the amount of funding and high-profile people involved, probably limp along for years to come). Last I checked, Coursera already had fungus growing on it: something about, "usg-sponsored studies find that coursera is better than starving in Africa" [4] http://en.wikipedia.org/wiki/List_of_programming_languages [5] The corresponding lisp narrative feels 'organic'. Consider the SBCL (steel bank common lisp) compiler. It is descended from CMUCL python compiler and uses some of its code, changing it as circumstances require (and of course, any transformations that needed automation were but a tree traversal away). Naggum also had some related thoughts:
[6] This could be largely automated, but that is a discussion for another day. |
|
Holy wall of text @gabriel-laddel! I understand that you fully wanted to underline your points but I would
Those are your points right, or am I misunderstanding them/missing some? Package management was indeed not built in from the beginning. The current You can think of this as a very lazy static evaluation of needed packages Regarding the hash checking, the observation of no problems with it still
So yes, both those things are only 99.9% solutions, but they are gaining 9s For the last point, I think Thank you for posting your thoughts, I still believe that NixOS and On Sun Feb 08 2015 at 6:51:59 PM Gabriel Laddel notifications@github.com
|
Uh... yeah, that's totally all the points I made. Toodles. |
|
NB: I'm a Lisp guy who uses and likes NixOS, but isn't a big expert in NixOS. Minor remarks: uiop:run-program has a now reliably portable :directory clause. NixOS isn't perfect, but is accepting patches. If you're interested in static linking of libfixposix into sbcl images, I have recipes that need to be put together into actual code, that you may be interested in. If you want a NixOS package with dynamic linking the solution is to either enhance cffi so you can statically specify the search path, or have a wrapper that initializes the search environment around the binary. Also, the solution to false negatives is that a package breaks if it's missing a dependency, so whoever is writing the package specification will notice and fix it — and thereafter it will build deterministically. |
|
Thanks for the info Francois. FTR, NixOS is fundamentally braindamaged. I found it was easier to create my own distribution than to hack around their crud. Details here: http://gabriel-laddel.github.io/system.html On Fri, Jul 10, 2015 at 4:11 PM, François-René Rideau <
|
I'm leaving this here in hopes that it'll be useful to someone. I didn't see anything in the NixOS manual that would indicate that I'd run into this class of problems while using it (granted, I've not read it in detail, but it's largely unreadable and overall, the whole codebase is poorly documented - attempting to discern the intent of the programmers isn't worth my time anymore, eg, where is the BNF form for the Nix language? What was so wrong with existing languages that a new one had to be invented. Where are the design documents? Perhaps I'm just not looking hard enough... in any case, looking hasn't proved useful, and reading the sources has revealed only a complete and utter disregard for quality.).
Cheers.
The text was updated successfully, but these errors were encountered: