Why do we have to wait one month before updates ?

[vindarel]

Hi,

I wonder why we have to wait one month before package updates. It is annoying because it prevents me from using my own library, or showing code using it, when new features are not there. It is simply surprising from a package manager. This question is not new and can be seen as a serious limitation, for example the author of this post lists it as one of the reasons he's strongly considering to stop using CL (what he may not do it in the end), while giving context:

Zach Beane has done an excellent job with Quicklisp, and it is far better than what we previously had available, but still has a few problems. Zach puts a lot of effort into curating a list of compatible software together into the form of a “Quicklisp dist” rolled out approximately every month. While this is great and puts near-zero maintenance on developers, it poses a few problems.

Firstly, Zach is a single point of failure. Yes, anyone can maintain their very own Quicklisp dist, but it isn't going to see the masses, and the Quicklisp internals are not very well understood by anyone other than Zach.

Also the fact that the official dists are rolled out so far apart (a month or longer in the software world is an eternity), means developers cannot push hot-fixes or address user-reported bugs in a timely manner, without pushing maintenance onto the users by having them checkout upstream sources.

Modern languages such as Julia and Racket offer central repositories where a developer can register their software projects, and will be automatically indexed periodically, so that users can continue to install and update software without any maintenance, and still receive updates quickly when needed. Additionally, they push managing version dependencies onto the developer, which I do not believe to be a bad thing. In constrast, Common Lisp libraries are rarely versioned, and all of that maintenance is forced upon the Quicklisp dist curator.

Also more explanation and possible workaround:

Quicklisp provides more assurance than you might expect. The criterion for inclusion, and maintenance, of a project in Quicklisp is that the project successfully load in SBCL alongside all the libraries in that Quicklisp distribution, and load without any errors or warnings. SBCL has extensive type inference and can catch and warn about many potential issues at compile time. And because of the pervasive use of macros in CL, successfully loading a library usually exercises a lot of code paths in its dependencies. To a surprising extent, "if it loads, it runs."

Qlot isn't something a library would use. You use it to set up the dependencies for an application. My approach (with TBRSS
) is this. Obviously, every time I upgrade to a new Quicklisp dist, I run a test suite to make sure everything is working together. On the rare occasion there's a problem, I either pin the offending library to an older version (with Qlot) or I fork it and fix it, again using Qlot to pull from the fork until the fix makes it into the next Quicklisp dist. And of course I also use Qlot for dependencies that are not, for whatever reason, available in Quicklisp.

https://www.reddit.com/r/lisp/comments/6snw5d/questions_for_2017_common_lisp_experts/dljcuz9/

Will Quicklisp dists have short release cycles ?

Or will a library be able to push its changes immediatly ?

Is there plans to make the situation better ?

While we're at it, could you explain what is the work relying on you for a package update ?

Regards

[quicklisp]

A month between updates is a compromise between chaos and stability.

It is annoying because it prevents me from using my own library, or showing code using it, when new features are not there.

Nothing in Quicklisp prevents you from using your own library however you like. One easy option is to put its source code in ~/quicklisp/local-projects/ and update it as often as you need.

There is a problem when a buggy version of some project is in a Quicklisp release. On rare occasions, for widely-used projects, I have made quick releases to fix them. But for most cases, it is indeed a month to wait for a fix while you continue to use the old, working version.

I hope to make it easier for people to learn how to make their own dists. Then people will have the opportunity to make new software sets following the policies that are most important to them. I think there's also plenty of room for other package managers for Common Lisp - maybe something styled more like clbuild would suit people who want instant access to updates.

The work relying on me is that I build everything in Quicklisp to make sure they build together before making a release. This covers a useful class of bugs. I'd love to incorporate more tests in the process to catch release problems that don't manifest at build time.

[vindarel]

Nothing in Quicklisp prevents you from using your own library however you like. One easy option is to put its source code in ~/quicklisp/local-projects/ and update it as often as you need.

sorry, I meant using new features of the lib in another project, which pulls it from quicklisp.

But what is the rationale behind 1 month ?

I build everything in Quicklisp to make sure they build together before making a release.

I imagine this could be automated further, with more work, more manpower.

So do you have plans to shorten the 1 month delay, or must it not happen ?

ps: Hopefully with my questions and your answers you'll be able to enhance Quicklisp's doc ;)

[quicklisp]

I don't think shorter release cycles are an unqualified good. In my experience, short release cycles can lead to instability and unpredictability, and I chose one month as a balance between getting timely updates and having a reliable, stable base.

The most labor-intensive part of making releases is monitoring daily failures and reporting bugs to the right people. The daily failure report is automated, but reporting bugs (and following up) can be a slog.

[Hexstream]

I realize many people won't find this an interesting remark but my "platonic ideal" would be a much more distributed approach, with releases every week.

In any case, I think vindarel raises important points, and I really hope we won't be stuck with the current model (which is pretty much exactly the initial model?) forever.

(Probably forking territory?)

[quicklisp]

I don't think a fork is necessary in this case. quicklisp-controller is on github, and has everything you need to bootstrap your own dist, identical to the quicklisp dist, and update it as frequently or infrequently as you like. And it's pretty simple for anyone who likes to subscribe to that dist.

[phoe]

@Hexstream I don't think a fork will be required. I think this is less of an issue with @xach not willing to make changes required by the community and more of lack of manpower that's willing and able to take up the burden of building and maintaining an automated Quicklisp dist/repository.

[quicklisp]

Even with all the automation in the world, I think I would stick to a one-month release schedule. If I could wave my hand and make work happen, it would be to make it easier for people with other ideas to publish their own dists with the contents and policies they prefer. For example, I'd love to see a dist for LispWorks-only software - there is a nice ecosystem there. I'd also love to see a bleeding-edge dist that pulls the latest of everything. But the default will probably continue to look a lot like it looks now.

[cbaggers]

I have benefited multiple times from the 1 month release cycle and the due diligence that Xach and others put in every month. Even if they had a team of 50 I'd still vote to keep it as it is now

Making a separate dist is far more sensible if you need extra control over delivery times and it will still play well with quicklisp and your other projects.

[vindarel]

Also, if I know that some required features didn't make it into quicklisp yet, can I not configure package sources, to pull a project from a tag, a release, the latest ?

On Quicklisp, something like

(ql:load-from :my-lib :latest) ;; actually just a shortcut for a git clone in ~/quicklisp/local-projects/

and more importantly in a .asd:

  :depends-on (:dexador)
  :depends-on-version ((:str :latest)
                       (:plump "http://…github/realease, tag, branch")))

that way I have an alternative to wait a month and I can deliver. One month later, I can change this.

---

quicklisp-controller is on github, and has everything you need to bootstrap your own dist, identical to the quicklisp dist, and update it as frequently or infrequently as you like.

That's the other side of the coin. If I just want to update a bugfix of my lib in Quicklisp, how do I do ? This use case is a limitation of the current process, AFAIU.

Quicklisp-controller

The software to build Quicklisp dists.

https://github.com/quicklisp/quicklisp-controller

The Quicklisp Controller is responsible for building and updating Quicklisp distributions from a project list (maintained separately in the quicklisp-projects repo). It fetches project source from a variety of places: git, cvs, svn, http-available tarballs, etc, and uses that to build up metadata about the relationships between projects. New or updated projects are then uploaded into the quicklisp download space.

[quicklisp]

Regarding pulling from git in the client, Quicklisp doesn't work like that. I did not want to rely on external processes in order to be portable to all Common Lisp implementations and platforms. This was a real issue in clbuild and asdf-install.

If you want to update a bugfix of your library, fix it, and wait a month.

[akacase]

I'm with @xach on this. If we want a rolling channel or something with quicker updates, we should setup a different dist that follows a more "nightly" release cycle and shoulder the burden for hosting.

[svetlyak40wt]

@vindarel instead of

Depends on version in asd, did you try a Qlot?

It allows to pin library version and even to pull the latest or tagged library from git.

[salva]

As somebody which uses both CL and Perl I would say that I find the CPAN approach way better than the QuickLisp one in that regard.

The main difference been that the author is in charge of when releases for his packages are done, so that, for instance, the lapse between somebody reporting a bug and a new release with a fix being available may be typically a couple of days or just hours. It also allows the author to tag releases as development or stable, so that users can choose to use one or the other easily.

[phoe]

This thread is turning into a discussion about "why X approach is better than Quicklisp approach" which leads to a fruitful, but dead point.

If someone is not satisfied with the way the current Quicklisp dist works and would rather have a more automated solution, then they are free to extend the quicklisp-controller to their liking and implement the required functionality for:

• creating a centralized service that acts as a QL repository and dist manager,
• allowing the authors to upload and/or update their projects on that service,
• setting up a CI test loop on that service for verifying that the packages build,
• automatically updating the service's dists with new releases,
• maintaining all of the above.

Until such a person or group of people appears, nothing is going to be be achieved and nothing in Quicklisp is going to change.

Talk is cheap - @xach has at least built something that works and can act as a foundation.

[vindarel]

@svetlyak40wt I saw Qlot but didn't investigate into how to integrate it into a workflow yet. Thanks for the new suggestion.