Use XDG base directories, if available

[tapeinosyne]

This PR adds basic support for XDG base directories; specifically, it makes XDG_CONFIG_HOME/sheldon the default root folder for Sheldon if $XDG_CONFIG_HOME is defined. If desired, the PR could be easily updated to make appropriate use of XDG_DATA as well as XDG_CONFIG.

There is no change in functionality: it's just a simple way to help users keep their home folders tidy.

Two notes on the patch:

• when determining Sheldon's root folder,XDG_CONFIG_HOME is simply ignored if it is not valid Unicode. Do we want to alert the user and abort instead?
• I am not too clear on the purpose of Sheldon's --home flag, which is why this code still aborts if XDG is set but a home folder cannot be found. Does home serve a purpose beyond providing a prefix for the default root? In which cases would a user pass --home rather than --root?

[rossmacarthur]

Hi @tapeinosyne, thank you for your contribution! I think this is a good improvement.

--home

I am not too clear on the purpose of Sheldon's --home flag, which is why this code still aborts if XDG is set but a home folder cannot be found. Does home serve a purpose beyond providing a prefix for the default root? In which cases would a user pass --home rather than --root?

Right now the home directory is used for the following things:

• Figuring out the default root directory.
• Replacing the home directory with a tilde (~) in the UI output.
• Expanding tilde's in paths in the config file.

--home would only be used if you were running on some unknown Linux where the home package could not figure out the directory. This is highly unlikely.

XDG Base Directory

In the docs it specifies the following:

• XDG_CONFIG_HOME
  Where user-specific configurations should be written (analogous to /etc). Should default to $HOME/.config.

• XDG_CACHE_HOME
  Where user-specific non-essential (cached) data should be written (analogous to /var/cache). Should default to $HOME/.cache.

It seems to me that it might make more sense that if we used the cache directory as well?

The current default folder structure is like the following.

$HOME/.sheldon    # root / cache_dir?
    plugins.toml  # config_file
    plugins.lock  # lock_file
    repos/        # clone_dir
    downloads/    # download_dir

I think that plugins.lock, repos/ and downloads/ are more like cache information. So perhaps we should keep those in the the $XDG_CACHE_HOME/sheldon directory.

If XDG_CONFIG_HOME then we could have a folder structure like this.

$XDG_CONFIG_HOME/sheldon  # root
    plugins.toml          # config_file

$XDG_CACHE_HOME/sheldon   # cache_dir?
    plugins.lock          # lock_file 
    repos/                # clone_dir
    downloads/            # download_dir

What do you think?

[tapeinosyne]

It seems to me that it might make more sense that if we used the cache directory as well?

It certainly would.

I think that plugins.lock, repos/ and downloads/ are more like cache information. So perhaps we should keep those in the the $XDG_CACHE_HOME/sheldon directory. What do you think?

I myself feel a degree of uncertainty over the difference between "essential" XDG_DATA and "inessential" XDG_CACHE. For repos and downloads, however, I would be inclined to favor DATA, ultimately yielding this folder layout:

$XDG_CONFIG_HOME/sheldon  # root (although the term loses much of its meaning)
    plugins.toml          # config_file

$XDG_DATA_HOME/sheldon    # data_dir
    repos/                # clone_dir
    downloads/            # download_dir

$XDG_CACHE_HOME/sheldon   # cache_dir
    plugins.lock          # lock_file 

The Debian Wiki entry might help understand my reasoning. Consider:

• Is the data/cache meant to to persist, e.g. across reboots?
• Is the data/cache used by Sheldon and Sheldon alone?
• What happens if the data/cache is wiped? Does something break?
• If wiped, how expensive is it to recreate—losslessly?

plugins.lock is arguably a cache, in that it is plainly derived from user and system configuration, and used only internally by Sheldon; generating it takes a fraction of a second. Whether the user knows it exists or not makes little difference, and mistakenly wiping it causes no real trouble.

The repos and downloads folders, on the other hand, can not be deleted without issue. Deleting them means that a new shell will block, awaiting a full redownload, which would qualify as "breaking", however remediably. So, here I'd personally favor DATA, as the user-specified repositories are more or less essential for sheldon to work as expected, without interrupting a user's workflow.

One last thing

There is another small decision to make: I reckon it should be reasonable to assume that, if any XDG_ base directory is defined in the environment, the user will want sheldon to follow the XDG folder structure as specified and use the XDG defaults for undefined XDG_* variables. "Reasonable", however, means neither necessary nor certain. Would this behavior suit you?

[rossmacarthur]

I myself feel a degree of uncertainty over the difference between "essential" XDG_DATA and "inessential" XDG_CACHE.

Yeah I agree. I'm happy with your proposed directory structure, although it might be simpler to just put plugins.lock in $XDG_DATA_HOME/sheldon as well, but I'm fine either way.

Implementation details

• I think it makes sense to use an XDG directory structure if any XDG_ prefixed environment variable is present.
• I think lets do away with the concept of a root directory in Settings. We can now have a config_dir and data_dir which in the absence of an XDG structure would both default to $HOME/.sheldon.
• Currently the root directory is used in three kinds of places which will have to be changed:
  • Sheldon uses this directory for the File mutex that is acquired. I think we just use the config directory, config_dir, for this instead.
  • In template rendering it is available as {root}. I think this can be removed. Perhaps later other directories can be made available in templates instead.
  • In clean() it is used to check whether the clone directory or downloads directory start with this path. This one would obviously now map to the data_dir.

Tests not passing

You should rebase your branch on the latest master, there was some weird things with linking when cross compiling which are resolved now on the latest master.

[tapeinosyne]

it might be simpler to just put plugins.lock in $XDG_DATA_HOME/sheldon as well, but I'm fine either way.

Yes, I thought about it as well. Even though using XDG_CACHE is probably, narrowly the most compliant approach, I frankly agree that it is overzealous to relegate a single file to its own lonesome folder on the basis of a probably, narrowly. I'll let plugins.lock into XDG_DATA.

I think lets do away with the concept of a root directory in Settings.

Agreeable. Let's.

Sheldon uses this directory for the File mutex that is acquired. I think we just use the config directory, config_dir, for this instead.

It might be best to lock plugins.lock as well, I reckon.

In template rendering it is available as {root}. I think this can be removed. Perhaps later other directories can be made available in templates instead.

That would be a breaking change, correct? It's hard to judge how many people might be using {root} — not many, I would wager? — but even so, I would be happy to consider some kind of deprecation if you'd rather take a cautious approach. Mind, that work might be better left to a follow-up PR.

You should rebase your branch on the latest master.

Absolutely. The amended commit will be rebased as a matter of course.

[rossmacarthur]

It might be best to lock plugins.lock as well, I reckon.

The mutex is also used during any config edit commands (add, remove, edit) which can happen in the event that plugins.lock doesn't exist. Additionally, during the lock and source commands plugins.lock might not exist yet at the time the file mutex is acquired. Having more than one lock makes things complicated and could cause deadlock if one command tries to acquire two locks ever.

That would be a breaking change, correct? It's hard to judge how many people might be using {root} — not many, I would wager? — but even so, I would be happy to consider some kind of deprecation if you'd rather take a cautious approach. Mind, that work might be better left to a follow-up PR.

Yes I am aware, I was thinking we would have to bump to 0.6.0 for these changes anyway because we are changing logic based on XDG_ prefixed environment variables which might already be in peoples environments. If we are to preserve the {root} in templates, lets make it map to data_dir. Additionally, I think this commit of mine 10c64a3 already requires the next release of Sheldon to be breaking.

[tapeinosyne]

The commit for XDG compliance has been amended to follow the revised folder structure. (With apologies for the push -f spam—GitHub really ought to let contributors remove those messages.)

I have left out the removal of root for now, as it touches on quite a few other things. I reckon it might be best to open a dedicated PR, but have no issue with simply adding a commit here if you prefer.

[tapeinosyne]

XDG directory support is now covered by integration tests rather than unit tests. (The relevant commit can be squashed/rebased as you prefer prior to merging.)

I have also taken the opportunity for a minor refactor, which will be needed in some fashion come the deprecation of root.

[rossmacarthur]

The integration tests look great! Thanks for the hard work. Just one more small comment, but otherwise I think we good to go.

[rossmacarthur]

Hmmm, unfortunately because Sheldon's output is not ordered, the test case doesn't produce consistent output and thats why its failing on some builds.

[tapeinosyne]

I suppose we could test only one of repos and downloads, for now, or test them separately; they are handled by much the same logic in any case.

[tapeinosyne]

The directory structure for data_dir is now tested incrementally to avoid unordered output while verifying that both repos and downloads behave as expected.

[rossmacarthur]

Great work! Thanks @tapeinosyne 🙌

[tapeinosyne]

Excellent. I shall move on to the deprecation of root, then.