Request: option to specify alternative path to elpa directory

[tkf]

... or environment variable, if it is easier.

If we have this, we can run test with multiple Emacs versions in parallel.

[rejeep]

So:

carton [...] --elpa-dir path

Maybe we should add a .cartonrc file as well...?

[tkf]

How is .cartonrc different from Carton? I'd like to be able to do this without changing a file. So I prefer option or environment variable.

[rejeep]

What I mean is that you can add a .cartonrc file for your default options. For example if you always want to use another path than elpa, you can have a config file, like this:

--elpa-dir other-path

But let's skip that for now. An option is a great idea. Do you have any name suggestion? :)

[tkf]

I'd use --package-dir or --pkg-dir.

[rejeep]

Actually, the variable that controls this is called package-user-dir, why not use --user-dir or/and --package-user-dir.

[swsnr]

@rejeep @tkf Should Carton rather use versioned elpa directories by default? Such as "elpa-${emacs-version}"?
Imho that would make more sense than having to remember and explicitly choose the proper package directory.

[rejeep]

@lunaryorn That would mean that you have to run carton install if you update Emacs to another version. No biggie, but still needs to be done.

[swsnr]

@rejeep Probably this shouldn't be the default anway. I do not think that this is actually necessary for most packages, since the byte code should be fairly compatible within one Emacs series. I have been testing Flycheck against 24.2 and snapshot builds from a single elpa directory without troubles.

If the behavior is explicitly enabled with a special directive (such as (versioned-package-directories)), the user is aware of this, so I do not see this as a problem.

[rejeep]

Agreed, all in favor say ayyy!?

[swsnr]

@rejeep Agreed.

[tkf]

That would mean that you have to run carton install if you update Emacs to another version.

You should. Byte-compiled files are not compatible for different Emacs versions. Incompatibility could happen even if you change some Emacs Lisp macros, without changing Emacs core in C. So, there is absolutely no guarantee that packages compiled by Emacs x.y work correctly with Emacs x.(y + 1). See here for example.

I suggested to add an option because I thought @rejeep did not like the idea due to the discussion in #3. But yes, I think versioning elpa directory by default is better. FYI, similar Python tool uses directory .tox/pyXY for python version X.Y. So I suggest using something like elpa/X.Y (or even .elpa/X.Y) instead of elpa-X.Y, as I don't want to see a lot of directories in my project root and .gitignore will be simpler.

[swsnr]

@tkf There is no guarantee for sure, but practially there is no problem either, if you stay within the same series. Within a major version, Emacs Lisp macros usually don't change in backwards incompatible ways. They can't really, or every user's compiled init file would break.

As said, I didn't experience any troubles. If you're in doubt, you can also just delete elpa and let Carton rebuild it before running your tests.

That said, I like the directory scheme you proposed, as long as it is optional.

[tkf]

I had trouble when updating from 23 to 24 and I helped users suffered the same problem. If there could be a trouble, we should avoid it. It is for test suite. If wasting HDD solves problem then we should waste HDD. I don't want to waste my time for debugging errors because carton mixing versions and then notice that I should just reinstall everything.

I find no reason to use elpa/PACKAGE/ or elpa/X.Y/PACKAGE/ depending on option. If you are using just one version of Emacs then carton just uses elpa/YOR-EMACS-VERSION/PACKAGE/. How bad is that?

[swsnr]

@tkf Like I said, within a major version there is usually no trouble. Even the Emacs developers themselves do not expect trouble, or they would have used a version-dependent default for package-user-directory. Emacs 23 support is a different story.

Many users, me including, use Carton to manage packages in .emacs.d. Carton explicitly supports this use case, and there is even a Carton frontend named pallet for this purpose.

Using versioned package directories by default breaks this use case and forces users to explicitly configure a version specific package-user-dir in their init.el.

[tkf]

I wasn't arguing that we should have versioned package-user-dir under .emacs.d. That's nothing but harm, I agree.

Does carton know if it is working on .emacs.d or on developmental version of a project? If so, using versioned directory for the latter case makes everyone happy, right?

[swsnr]

@tkf I prefer an explicit solution over an implicit one. I think we should refuse the temptation to guess the user's intent. That's bound to break sooner or later.

What stands against an explicit directive? Even if we try to guess, we must provide one to allow users to explicitly override our guess in case we guess wrong.

[tkf]

In development, using the same package-user-dir for different Emacs versions is just a stupid idea. Why not avoid it by default?

Also, having different structure under elpa (i.e., does it have Emacs version part) confuses Emacs Lisp developers, especially when you are contributing to other people's package.

But anyway, having it optional or not is not a big deal at all. Even having just --package-user-dir is OK for me. I think I wrote my opinion above in order to make carton interface more concise. I am happy as long as there is a way to differentiate the location of package-user-dir.

[swsnr]

@tkf I know, but I am not sure whether we can safely differentiate between the ~/.emacs.d and the development use case by just guessing.

[rejeep]

I don't see why not always use .elpa/X.Y/package for both package development and your Emacs config? We should always prefer to handle them in the same way. Is there a downside other than a carton install when you upgrade Emacs major/minor version that I don't see?

I'm not sure we should allow the whole path to be configured, that would only be confusing. What could be configured though is the base directory, which in this case would be .elpa.

But I don't see why anyone would ever want to change the base dir? Not a good one anyways. Maybe we should store everything under the Carton directory instead, so:

.carton
  elpa
    drag-stuff
      24.1
        package-x
        package-y
      24.2
        package-x
        package-y
      24.3
        package-x
        package-y
    wrap-region
      24.1
        package-a
        package-b
      24.2
        package-a
        package-b
      24.3
        package-a
        package-b

[tkf]

I don't see why not always use .elpa/X.Y/package for both package development and your Emacs config?

I think @lunaryorn's point is that it is better to keep it as same as what package.el does by default. But I'd say versioned ~/.emacs.d/elpa/... is better than the default, as it enables you to use different Emacs versions (if you don't compile ~/.emacs.d/init.el etc.) I am not sure if it is good or not, as I don't use package.el directly (I use el-get). But it would be very unique feature, as el-get nor package.el does not allow you to use multiple Emacs versions.

Maybe we should store everything under the Carton directory instead, so:

You mean under ~/.carton/? Oh please don't do this! You can have many local clones. You may be using different versions of dependency. Keeping everything under your repository's working directory is better approach.

[rejeep]

I think @lunaryorn's point is that it is better to keep it as same as what package.el does by default. But I'd say versioned ~/.emacs.d/elpa/... is better than the default, as it enables you to use different Emacs versions (if you don't compile ~/.emacs.d/init.el etc.) I am not sure if it is good or not, as I don't use package.el directly (I use el-get). But it would be very unique feature, as el-get nor package.el does not allow you to use multiple Emacs versions.

Yeah, better than default. On my OSX machine, Emacs 22.1 is installed by default, but I of course install Emacs 24. That would allow me to use both if I wanted.

@lunaryorn What do you think?

You can have many local clones. You may be using different versions of dependency. Keeping everything under your repository's working directory is better approach.

Multiple repos? That's what branching is for! But don't worry, I have no plans to switch to that approach, just mentioned it :)

[swsnr]

@rejeep Currently, Carton works well for .emacs.d because the directory it installs packages to happens to be the same as the default value of package-user-dir. If we change the directory, our users have to configure package-user-dir explicitly in their init.el. carton-setup won't do, because the carton package itself isn't available until package-user-dir points to the right directory.

I don't mind such a change, for I find no difficulties in changing package-user-dir in my config, but other users may not find this simple.

But I came to think that you are both right. We should choose not to care now. Just using versioned directories by default everywhere keeps the code base simpler, and the user interface more predictable. Moreover, I like the directory hierarchy you proposed.

Putting all Carton-related data into a .carton directory in $PWD keeps the directory clean, and allows for future extensions, should we have need to save more metadata than just packages.

So let's just go for versioned directories. But we must document this changes, and we really should provide example code in README that explains how to properly configure package-user-dir when using Carton.

[rejeep]

carton-setup won't do, because the carton package itself isn't available until package-user-dir points to the right directory.

I don't follow. Why is there any difference between an Emacs package and the local Emacs installation?

[swsnr]

@rejeep I do not start my my Emacs through Carton, i.e. via carton exec. Thus Carton cannot setup the load path and the package directory. I need to do so myself in my init.el.

[tkf]

I guess @lunaryorn installed carton via package.el?

[tkf]

Re .carton: I think what @rejeep has in mind is $HOME/.carton and what @lunaryorn has in mind is $PWD/.carton. I think $PWD/.carton/elpa/EMACS-VER/PACKAGE is good, but I think $HOME/.carton/elpa/PROJECT/EMACS-VER/PACKAGE is a bad approach.

Multiple repos? That's what branching is for!

I find having local clones is better approach when I am working on more than three branches. Anyway, I think you should not assume people's workflow. Having local clones is valid usecae and one of the strength of DVCS such as git.

[swsnr]

@tkf I agree. A global Carton directory is just a bad idea, for any workflow.

[rejeep]

Yeah, I agree too. $PWD/.carton/elpa/EMACS-VER/PACKAGE is king.

You mean like this? https://github.com/rejeep/emacs/blob/master/init.el#L32-L33

[tkf]

@rejeep So, in your setting, you called (package-initialize) at L18, which is before (require 'carton). when (package-initialize) is called, package-user-dir is the default ~/.emacs.d/elpa, I suppose.

[swsnr]

@tkf Exactly. For package-initialize to pick up the Carton packages, you must set package-user-dir to the proper versioned directory before calling package-initialize.

This is not a show stopper, but it definitely must be documented and explained in the README.

[rejeep]

Yeah, for sure. I think it's worth it!

[tkf]

Does it mean all Carton users must have something like

(setq package-user-dir
      (locate-user-emacs-file
       (format "elpa/%d.%d" emacs-major-version emacs-minor-version)))

in their setting? Sounds bit sad.

Or can we do this other way around? Let user install carton in ~/.emacs.d/elpa/carton-X.Y (at this point, package-user-dir is the default ~/.emacs.d/elpa). Then, after carton is loaded, calling (carton-setup user-emacs-directory) resets package-user-dir to something like ~/.emacs.d/.carton/elpa/EMACS-VERSION and re-initialize package.el. I am not familiar with package.el so I am not sure if it works properly, though.

[tkf]

Re-pinging @lunaryorn and @rejeep. I see two options to use carton in init.el. What do you think? Which one is better?

1. Ask user to add the above (setq package-user-dir ...) in their init.el.
2. Install carton to ~/.emacs.d/elpa/carton-X.Y then other packages in ~/.emacs.d/.carton/elpa/EMACS-VERSION.

[rejeep]

Or start Emacs with carton exec emacs ...

[tkf]

You mean cd ~/.emacs.d/ && carton exec emacs. I don't mind but I guess people prefer if emacs just works.

[rejeep]

Yes, that's what I'm thinking. We could add that as an option to the README at least.

[tkf]

I think cd ~/.emacs.d/ && carton exec emacs should work even if we choose one of the option I described above. So, how about first make cd ~/.emacs.d/ && carton exec emacs work and then add an elisp interface (which can be called from init.el) to make plain emacs work?

[rejeep]

Sounds like a good idea!

[swsnr]

@tkf @rejeep I mean no offense, but this is just a bloody stupid idea. Do you seriously expect me to type cd ~/.emacs.d/ && carton exec emacs every time I want to use Emacs?!

This way to start Emacs is ridiculously long, messes with the working directory and can only be used from the terminal. It breaks, if someone tries to give a relative path name to Emacs, or wants to start Emacs via the graphical desktop environment.

If we are to use versioned directories in Carton by default, we must make plain emacs works. Otherwise the whole ~/.emacs.d/ use case goes down the drain…

[rejeep]

@lunaryorn You could make an alias. But this is just an alternative method.

If we are to use versioned directories in Carton by default, we must make plain emacs works. Otherwise the whole ~/.emacs.d/ use case goes down the drain…

Agreed. Maybe I misunderstood you @tkf. When we merge this change to master, everything should be working. What I thought you meant @tkf was that we make carton exec emacs work in it's own branch. And then, when we have that working, we add the Elisp interface and then merge to master. That is one step at a time...

This feature will include a DSL change for people using Carton in Emacs today and we only want them to migrate once, no twice.

[tkf]

I opened a PR #40. It is in @rejeep's repository so we all can push to it. Ecukes tests pass locally. #39 must be done first so that we don't need to fix unit test.

We still need to decide what to do to make plain emacs work.

[rejeep]

We still need to decide what to do to make plain emacs work.

For #2, the user still have to add Carton to load-path. If Carton would have been part of Emacs and we would not have to care about that, then #2 is better of course. But since the user have to add custom code either way, I think #1 is better.

[tkf]

What I meant by #2 was something like this:

(package-initialize)                     ; [1]
(require 'carton)
(carton-setup user-emacs-directory)      ; [2]

At [1], package-user-dir is ~/.emacs.d/elpa. After [2], package-user-dir is ~/.emacs.d/.carton/elpa/EMACS-VERSION. package-initialize is called again in [2]. But I noticed you can't update carton using package.el so this is out, I guess.

Other idea is to add this in init.el: (load "~/.carton/carton.el")

If we go to #1, there is actually no fix is needed in carton.el. Just some README tweaks.

[rejeep]

Other idea is to add this in init.el: (load "~/.carton/carton.el")

That's not too bad, actually. We could also do: (require 'carton "~/.carton/carton.el")

If we go to #1, there is actually no fix is needed in carton.el. Just some README tweaks.

Right, the only thing affected are the users.

[tkf]

User can choose from #1 or "explicit" require. They are both usable after #40. If @lunaryorn agree, I will merge #40. Or do you guys want to use it for a while before merging it?

[rejeep]

The changes to Carton are pretty straight forward, so I'm not that worried that the code won't work. But before we merge to master we first need to update the README. I'm also thinking about deprecation. If we merge #40 right now it will break on a lot of Emacs users machines. I'm not sure how to best go about this. We need a way to notify users and not break before they have migrated.

[tkf]

How about doing this when changing the project name? We can just warn people that the project is updated and the configuration is a little bit different.

[rejeep]

Yes. I think we should create a branch (v0.4-wip) were we can merge these edge features: name switch, bin/carton in Emacs Lisp, versioned directories, etc... And when we feel that branch is ready, we release it, tweet, blog etc.. It's better we do one big change than multiple.

[tkf]

v0.4-wip sounds good.

Here is the new README: https://github.com/rejeep/carton/tree/versioned#local-emacs-installation (diff: 1aa3ce7)

[rejeep]

Nice. I made a minor document comment on the PR.

I created an issue for v0.4 (see #42). If you can think of anything else, just add it to the issue, if you can edit? Otherwise, comment and I will add it.

So, now you could create the v0.4-wip branch and merge versioned to it. And also check that point in #42.

[tkf]

Thinking about the path, maybe .carton/EMACS-VERSION/elpa is better than .carton/elpa/EMACS-VERSION. This is how tox works. We may want to have .carton/EMACS-VERSION/tmp for test. If FFI support for Emacs is in, we may even need to have .carton/EMACS-VERSION/include and .carton/EMACS-VERSION/lib. It's not serious problem, but .carton/EMACS-VERSION/elpa seems to be more extensible.

[rejeep]

@tkf I don't mind that.

[swsnr]

@tkf Good idea.