Building stable packages using VC tags

[milkypostman]

When a recipe specifies an upstream source with numeric tags (or perhaps branches), we can use that info to build stable packages.

package.el won't easily handle coexistence of stable and snapshot packages in the same package archive, since package-list-packages (currently) only shows the highest available version number of a package, and package.el provides no way to selectively install a version 1.4 instead of 20120406.

Renaming snapshot packages to add a -snapshot suffix is an option, but might wreak havoc with inter-package dependencies; foo-snapshot packages would not satisfy a dependency upon foo, for example, and so dependency names might need to be updated.

The alternative plan we've discussed is to provide two parallel archives, each with separate URLs and contents; one would contain snapshot packages, and the other would contain stable packages. In reality, though, many users would want to combine packages from both archives, and so machinery like that provided by melpa.el would be required in order to selectively install stable package versions when snapshots are also available.

[haxney]

Having used Ruby, Rails, and Bundler a fair amount recently, I think the Ruby world essentially has it right on this issue. My recommendation is to copy them shamelessly, since mixing Git snapshots with stable packages is relatively painless in that world.

[technomancy]

I think it would be best to add support for prerelease versions to package.el itself. It may still make sense to keep them in separate repositories, (rubygems and maven both do this) but you want the default to be only to pull in releases; snapshots should only come in when explicitly requested.

[purcell]

Hey, @technomancy, what could package.el learn from nix...?

[milkypostman]

how to have a name that's hard to search for...

[technomancy]

@purcell Unfortunately I think the nix approach can't really be applied to Emacs right now. Isolation of code is really impossible in elisp, so the notion of immutable trees of dependencies really can't be applied. So the primary lesson is just that the lack of a module system is a really crippling flaw.

[haxney]

I especially like Bundler's ability to list dependency versions like ~> 1.2.3 which means it depends on version V being 1.3.0 > V >= 1.2.3. That goes arbitrarily deeply:

• ~> 1.2.3.4 == 1.2.4.0 > V >= 1.2.3.4
• ~> 1.2 == 2.0 > V >= 1.2

Re: Nix: it's a neat take on packaging. package.el already uses things like per-version directories for libraries (with a version number rather than a sha1). Elisp is mutable and dynamically scoped by design, so there is not much to gain from looking at purely-functional systems.

[technomancy]

I think the main thing that's actually feasible that's missing is the ability to separate out prerelease versions from actual releases. Rubygems does a decent job at this; it will only look for prerelease versions if the user explicitly asks for it.

I would also like the ability to specify exact version dependencies instead of minimum dependencies, but I think that would complicate the dependency resolution logic by several orders of magnitude, and I'm not sure it's worth it given that elisp isn't used for production software. (so repeatability is not of paramount importance)

[milkypostman]

so, can we just add a configuration tag here called :stable or something that is either a branch name or a tag regex? Then we will build two versions, if we can extract a version from the tag regex then we use that, otherwise we use the hardcoded version? I think we can just make it simple at first then expand for different requirements.

[purcell]

@milkypostman You mean build both versions into the same archive?

[milkypostman]

no, i mean, just build two packages each run. and have two package archives.

On Mon, Jun 17, 2013 at 1:08 PM, Steve Purcell notifications@github.comwrote:

@milkypostman https://github.com/milkypostman You mean build both
versions into the same archive?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7#issuecomment-19571103
.

[purcell]

Yeah, to accomplish a dual build without a ton of hackery, I think we should basically run the whole build top-down in two different modes, setting a different output dir for each using the current defvars. And a new defvar would be used to toggle the build mode.

I'm not sure about "either a branch name or a tag regex" -- the tag regex I can understand, but a branch name isn't sufficient to identify both a specific commit and a name for the release, since any number of commits can be made on a branch. Ideally we'd have a standard tag regex, because otherwise we'd presumably end up repeating a very similar pattern in all the recipes.

[milkypostman]

but isn't is possible that someone could just consider a branch to
determine its own version via the *-pkg.el file or in the main .el file via
the Version: commentary? I can just see some people wanted to have
"stable" branch rather than a tag. But i'm ok with us sticking to the tag
idea.

On Mon, Jun 17, 2013 at 1:38 PM, Steve Purcell notifications@github.comwrote:

Yeah, to accomplish a dual build without a ton of hackery, I think we
should basically run the whole build top-down in two different modes,
setting a different output dir for each using the current defvars. And a
new defvar would be used to toggle the build mode.

I'm not sure about "either a branch name or a tag regex" -- the tag regex
I can understand, but a branch name isn't sufficient to identify both a
specific commit and a name for the release, since any number of commits can
be made on a branch. Ideally we'd have a standard tag regex, because
otherwise we'd presumably end up repeating a very similar pattern in all
the recipes.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7#issuecomment-19573053
.

[milkypostman]

OK lets start with melpa/powerline I have tagged a version 2.2.

Lets assume that the default way to do this is a regex that is ([[:digit:].]+) ?

[purcell]

Yes, ideally we'd want to support any versions supported by version-to-list. But in VC, there will often be a v or version- prefix, or (in CVS, for example) the . might be replaced by a -. (True flexibility would mean having combinations of patterns plus group indices, a la compilation-error-regexp-alist-alist, but that would hopefully be unnecessary)

So a simple first step would be that we'd add a :stable t to the powerline recipe, and a package-build-stable boolean var defaulting to nil, and then look at what code needs to be changed such that a full build with package-build-stable set to t would spit out just powerline.

I imagine we'll need per-SCM functions for querying tags, and we'll also need to have per-SCM code for checking out specific tags: in the case of subversion, for example, this will likely require the inclusion of additional information about the repo structure in the recipe -- knowing a URL like "...foo/bar/trunk" is not sufficient to locate tagged versions in that same repo.

Starting with git is a reasonable first step.

[milkypostman]

couldn't we just assume for the time being that if we can find a tag in the
current repo that we should build a stable version? I don't think manually
enabling for each package is necessary (i.e., with a stable t line).

for svn, couldn't we assume if the repo ends with trunk to look in the
parent directory for tags? I think we're definitely going to have to
worry about places where versions are separated by underscores or dashes.
Hadn't thought about that.

On Tue, Jun 18, 2013 at 6:54 AM, Steve Purcell notifications@github.comwrote:

Yes, ideally we'd want to support any versions supported by
version-to-list. But in VC, there will often be a v or version- prefix,
or (in CVS, for example) the . might be replaced by a -. (True
flexibility would mean having combinations of patterns plus group indices,
a la compilation-error-regexp-alist-alist, but that would hopefully be
unnecessary)

So a simple first step would be that we'd add a :stable t to the powerlinerecipe, and a
package-build-stable boolean var defaulting to nil, and then look at what
code needs to be changed such that a full build with package-build-stableset to
t would spit out just powerline.

I imagine we'll need per-SCM functions for querying tags, and we'll also
need to have per-SCM code for checking out specific tags: in the case of
subversion, for example, this will likely require the inclusion of
additional information about the repo structure in the recipe -- knowing a
URL like "...foo/bar/trunk" is not sufficient to locate tagged versions in
that same repo.

Starting with git is a reasonable first step.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7#issuecomment-19612476
.

[purcell]

That would be nice, but it's not a safe assumption. For example, if an Emacs mode is part of a larger code repository (think Erlang), then does a repo-wide tag imply that the Emacs mode has that version? Usually not. So a safe scheme (at least initially) would be to opt in recipes one by one, or use some heuristic, e.g. whether the recipe includes files at the top level of the repo.

Similarly, the suggested heuristic for subversion tags based on common practice is worthwhile, but it won't be the full story.

[milkypostman]

dang you're always right!

On Tue, Jun 18, 2013 at 7:47 AM, Steve Purcell notifications@github.comwrote:

That would be nice, but it's not a safe assumption. For example, if an
Emacs mode is part of a larger code repository (think Erlang), then does a
repo-wide tag imply that the Emacs mode has that version? Usually not. So a
safe scheme (at least initially) would be to opt in recipes one by one, or
use some heuristic, e.g. whether the recipe includes files at the top level
of the repo.

Similarly, the suggested heuristic for subversion tags based on common
practice is worthwhile, but it won't be the full story.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7#issuecomment-19616121
.

[purcell]

Nope, my wife is always right.

[rejeep]

I would also like the ability to specify exact version dependencies instead of minimum dependencies, but I think that would complicate the dependency resolution logic by several orders of magnitude, and I'm not sure it's worth it given that elisp isn't used for production software. (so repeatability is not of paramount importance)

@technomancy Very complex indeed, but you are wrong about Elisp not being used in production. I use Emacs everyday for work. If I for some reason remove my ELPA bundle and install again, I most likely will get new versions installed, which may break my Emacs installation. This has actually happend to me a couple of times. Specially fun spending halv a day fixing that when you are soaked at work.

If we ignore the complexity I think MELPA should create one build per VC-tag. The problem with this is that package.el would either need to support multiple versions (see http://bzr.savannah.gnu.org/lh/emacs/emacs-24/annotate/head:/lisp/emacs-lisp/package.el#L153) or send a list of all packages including version to the mirror before syncing the archive contents.

I would very much like to be able to add support for a Carton.lock file when using Carton to make sure that you will always get the exact same code every time you bundle.

[purcell]

Well, more generally you'd want a package-locked-package-versions variable or similar, along the lines of the recently-added package-pinned-packages variable. Carton could then piggyback onto that where available.

[rejeep]

Yes, that would have been a simple solution.

[milkypostman]

soooo http://hiddencameras.milkbox.net/

only git and only packages that have a tag.

i need @purcell to school me on this sys admin stuff. it always feels like a hack, so now we have two sets of service scripts.

[purcell]

@milkypostman Yeah, we should perhaps even have 2 separate build users...

Great that you set that stuff up so we can see what it produces. Looks pretty good, doesn't it?

One immediate issue I see is the one that I foresaw about elisp hosted inside the repo for a bigger project: cfengine-code-style, for example, has picked up the version of the enclosing repo, "3.5.3".

A nice way to handle this might be to support upstream version tags in the form "package-name-1.2", so that package-name.el could be versioned independently of the rest of the repo. I wouldn't want to rely on a file's Version: header.

Also, the haskell-mode version is listed as 2.-3.9.-3.1... I wonder how that happened. scala-mode says 2.10.0.13.6, which is a bit crazy-looking.

Relatedly, @tarsius had mentioned that we should probably provide a way to exclude pre-release tags, and I think a recipe mechanism something like :ignore-versions "\\.90$" might be a reasonable way to do this.

[purcell]

In descending order of popularity, the other fetchers we'll need to modify appear to be: hg, svn, bzr, cvs and darcs. I don't imagine we'll ever be able to build reliably-versioned stable packages from code on the wiki, unless we just decide to trust the Version: header in that case.

[milkypostman]

For a first run it doesn't look so bad. Is there a problem with using the main repo version for the package version for large packages? It makes sense to me that when a new Scala version is released that the new Scala mode package have a corresponding version.

[milkypostman]

Also darcs before cvs man!
On Jan 26, 2014 3:20 AM, "Steve Purcell" notifications@github.com wrote:

In descending order of popularity, the other fetchers we'll need to modify
appear to be: hg, svn, bzr, cvs and darcs. I don't imagine we'll ever be
able to build reliably-versioned stable packages from code on the wiki,
unless we just decide to trust the Version: header in that case.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7#issuecomment-33314479
.

[purcell]

Is there a problem with using the main repo version for the package version for large packages? It makes sense to me that when a new Scala version is released that the new Scala mode package have a corresponding version.

The problem is that the elisp is often versioned separately with a Version: header, so if the author has published a stable "1.2" version to Marmalade, we probably shouldn't pop up and claim that "3.8.21" is the stable version just because we found that tag in the repo.

[milkypostman]

Oh yeah derp. Good point.
On Jan 26, 2014 12:55 PM, "Steve Purcell" notifications@github.com wrote:

Is there a problem with using the main repo version for the package
version for large packages? It makes sense to me that when a new Scala
version is released that the new Scala mode package have a corresponding
version.

The problem is that the elisp is often versioned separately with a
Version: header, so if the author has published a stable "1.2" version to
Marmalade, we probably shouldn't pop up and claim that "3.8.21" is the
stable version just because we found that tag in the repo.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/7#issuecomment-33330022
.

[technomancy]

The problem is that the elisp is often versioned separately with a Version: header, so if the author has published a stable "1.2" version to Marmalade, we probably shouldn't pop up and claim that "3.8.21" is the stable version just because we found that tag in the repo.

Why not just refuse to include packages in the tagged repo if they don't either 0) tag specifically for the elisp lib or 1) provide prefix under which elisp-specific tags can be found?

[purcell]

@technomancy That wouldn't cover the case where an elisp-only repo has its lisp in a "src" or "lisp" subdir: the entire repo would be tagged, not just the elisp dir. But yes, the tag prefix idea you mention is pretty much what I suggested in my earlier comment, and seems like a good way to go.

[milkypostman]

Closing this. I think we are doing this now. We can file further issues in the future if we need to track it.