OfflineIMAP does not follow XDG Base Spec

[gioele]

OfflineIMAP stores its configuration in ~/.offlineimaprc. Instead of cluttering the home directory, it should follow the XDG Base Directory spec and store its configuration in its own directory under $XDG_CONFIG_HOME (.config by default).

http://standards.freedesktop.org/basedir-spec/basedir-spec-latest.html

[konvpalto]

There are many standards, as usual. What gains do you see for following XDG specification for OfflineIMAP?

[gioele]

There are many documents around explaining why XDG Base matters. The most concise I've found is https://live.gnome.org/GnomeGoals/XDGConfigFolders.

BTW, XDG is the only starndard I know for storing preferences, data and cache files in Linux systems. What other standards are there? Also, as pointed out in https://bugzilla.mozilla.org/show_bug.cgi?id=735285 XDG has been adopted de facto by all the big coordinated projects.

[aspiers]

+1. This is the only standard I'm aware of in this area, and adoption is a no-brainer IMHO.

[gioele]

@aspiers also because OfflineIMAP can use http://www.freedesktop.org/wiki/Software/pyxdg/.

[aspiers]

@gioele Right. I think you meant to direct that to @konvpalto?

[konvpalto]

Well, another de-facto standard for Unix systems is to use dot-file. OI doesn't want any caches or data directories as per XDG, so we're talking only about configuration.

As for the explanations, the only one that is really applicable to the current status of OfflineIMAP is cluttering of $HOME. The gain for OI itself seems to be 0, so I'd say that this is low priority enhancement that adds the problem of config migration. It could be handled as just a simple automagical rename of configuration file by OI itself + warning to the user, though it could break some stuff like automated backups that are already in place.

So my thinking is to leave this to be open until someone (may be me myself) will write the needed support code or find any arguments against using XDG.

[konvpalto]

pyxdg isn't needed at all: support for XDG_CONFIG_HOME is just two lines,
_config_dir = os.environ.get('XDG_CONFIG_HOME', os.path.join(os.path.expanduser("~"), '.config'))
_config_file = os.path.join(_config_dir, "offlineimap", "rc")
and there is no need for an additional dependency on the 143 Kb of Python code for doing just that, at least I currently don't see any additional utilities in pyxdg that can be used in OI.

[konvpalto]

@aspiers Anything being a no-brainer doesn't warrant its inclusion to any software -- it leads to the bloated thing that becomes unmanageable over the time, so I am trying to figure out the real gain of XDGfying for OI and users.

[aspiers]

The real gains are clearly listed in the URL @gioele already pasted above. Here it is again:

• https://live.gnome.org/GnomeGoals/XDGConfigFolders

I'm not personally religious on whether pyxdg gets used or not, although funnily enough I had exactly the same discussion elsewhere very recently.

[chris001]

konvpalto: "OI doesn't want any caches or data directories as per XDG, so we're talking only about configuration."

I don't entirely agree. The "cached current status" is suitable for the XDG "cache" - it's an item that, after admin or user deletes it, OI is supposed to automatically rebuild it on the next run. Deleting cached current status is a powerful troubleshooting technique for end users and admins, and needs to be made easy with the XDG standard.

"As for the explanations, the only one that is really applicable to the current status of OfflineIMAP is cluttering of $HOME. The gain for OI itself seems to be 0, so I'd say that this is low priority enhancement that adds the problem of config migration."

The gain for the OI project itself is less support messages submitted by confused users and admins looking to troubleshoot their OI installations and don't know how to safely reset the cached current status, nor the config settings. So they put strain on the dev team with support messages, an unnecessary strain, since there is a widely adopted standard out there, and if OI adhered to it, they could simply clear cache and settings themselves without having to double check with the support mailing list out of fear of data loss and many many hours of downtime - without XDG, one incorrectly deleted folder, and 50 GB of mail can take 24 hours for OI to catch up with and sync.

[gioele]

Il 23/05/2013 13:04, konvpalto ha scritto:

pyxdg isn't needed at all: support for XDG_CONFIG_HOME is just two lines,
_config_dir = os.environ.get('XDG_CONFIG_HOME', os.path.join(os.path.expanduser("~"), '.config'))
_config_file = os.path.join(_config_dir, "offlineimap", "rc")

If you are going to do that, please rememeber to also check for the
presence of a legacy ~/.offlineimaprc or move the legacy config file
to the new XDG-compliant location.

Gioele Barabucci gioele@svario.it

[konvpalto]

@chris001

"cached current status" (CCS) is not a cache and it is not going to be "just rebuilt": on the next sync, when CCS is missing, OI will download all your messages from the remote folder, so you will lose all local modifications to the corresponding maildir. Please, look at http://blog.ezyang.com/2012/08/how-offlineimap-works/ for more verbose explanation why is it so. One can remove it if is becomes broken, but it isn't an item that will have zero impact on anything, but speed, so it has not pure cache semantics.

As for resetting the config: what's that? You can't live without proper OI configuration, so it should be present in some form. Thus, either you're using version X or version Y of your config. That can be translated to using .config/offlineimap/rc or .config-virgin/offlineimap/rc or just .offlineimaprc/.offlineimaprc-virgin, provided one had created the -virgin variant. How can XDG help here without having prior backups of a single configuration file?

"...without XDG, one incorrectly deleted folder, and 50 GB of mail can take 24 hours for OI to catch up with and sync": how XDG helps in fixing this? Such massive lossage due to a single folder removal, as I could imagine, may result from removal of stuff from repository's FolderValidity subdirectory. By definition, it is better not to touch anything inside that directory, be it inside XDG-specified directory or not.

[WhyNotHugo]

IMHO the "cached current status" isn't the sort of cache the XDG Basedir spec refers to. I belive it should be stored in XDG_DATA_HOME, since it's user data (and of some value).

Adding support for XDG_CONFIG_HOME benefits the user by resulting in a cleaner $HOME (more and more desktop applications are XDG Basedir spec, which is basically the de-facto standard by now).

If support for using XDG_DATA_HOME for the CCS is added, there's the benefit of a "default location for all metadata", making that configuration option optional. Not a HUGE benefit IMO, but nice-to-have.

There's not much code necesary to support both of these.
Are you accepting patches for this feature? I can see if I can implement it when I get a some free time. :)

[xnox]

I find it nice to support XDG out of the box for new software.... but migrating existing software is usually a pain and we will be stuck requiring to check for both locations and support one or the other. While XDG is nice, I am yet to see it widely adopted by typical packages that are present on servers. E.g.: .ssh .bashrc .gnupg .procmailrc etc...
As I find that offlineimap is more of a unix tool, than a linux desktop app. Thus I also think continuing to use dot files is fine.

[aspiers]

That's a head-in-the-sand approach :( Supporting both locations is trivially easy, and doesn't pose any time constraints on migration. I have 63 directories in ~/.config on my workstation, so whilst many high-profile programs haven't yet adopted it yet, this doesn't mean it's lacking in momentum. Pre-existing inertia is not a good justification per se for resisting progress.

Also I can't understand why you're talking about servers and find that offlineimap is more of a UNIX tool than Linux desktop app? I know many users of offlineimap, and every single one of them runs offlineimap on their desktop or laptop. Why would you bother running it on a server? If you did that, you might as well just contact the original IMAP server directly.

[gioele]

The XDG Base spec is used by

• Chromium
• LibreOffice
• all Qt 4 programs
• all GTK+ 3 programs (via dconf)

Actually the situation is reversed: the only famous open-source desktop program that does not use XDG Base spec is Firefox.

[WhyNotHugo]

Indeed, firefox, pidgin, ssh, and gnupg are the only non-adopters I can think of.
All the others are no-longer-updated software.

Firefox has stated over and over that they don't care much for linux-specific integration or separating different types of data.
Pidgin is already working on this.

It's really trivial to search for both locations, but there could also be a mayor offlineimap release that searched for data in the new locations, and that would solve most of the issue (people will most likely read about the changes before upgrading to a nex major version).

[Stebalien]

ssh and gnupg have refused to adopt it because quite a few third party tools expect to find ~/.ssh and ~/.gnupg. This is not the case with offlineimap.

[aspiers]

Right - I don't think it's appropriate for this issue to be closed when there seem to be more concrete reasons and people in favour of addressing it than ignoring it.

[chris001]

Agreed XDG base spec is worth following.

The big reasons are making backups and troubleshooting "just work" even when done by non-admins and by technical people who are not intimately familiar with the OI code and don't have the time to become OI gurus.

Backups: without XDG, you don't exactly know where "all of the OI config and data is located". The danger is missing some data, or taking way too much data and clogging up your backup drive. XDG solves this by making it abundantly clear exactly where all of the config and data for the app is located. So even a non-admin can grab a copy and back it up to a safe location.

Troubleshooting: without XDG, you're grabbing at straws in the dark if you want to "remove data or config from the picture and see what happens - to rule out corruption of data or config being the issue". With XDG, that troubleshooting method is both possible and guaranteed to do exactly what you expected it to do - remove the data or config and see how to code runs differently.

I know, the big gotcha is the "remove the config and see what OI does" - then OI won't work, right, I know. This brings us to my next point. OI needs a VERY SIMPLE "consumer-friendly" way of communicating to the user - "The OI config is blank - please enter the following 4 pieces of information as a bare minimum - hostname, port, username, and password, for both server A and B - for a total of 8 pieces of info - or, for power users, edit the config file for more exotic options: local maildir, ssl enable/disable, blinking lights, single threaded, idle, etc..."

[konvpalto]

I'll try to add XDG support after 6.5.5 will be out: the release is long overdue and XDG requires quite a bit of testing and thinking on handling corner cases, so it's not a kind of low-handing fruit.

[qguv]

Has there been any progress on this, specifically with XDG_CONFIG_HOME, @konvpalto? I'd be happy to lend a hand with a pull request if there's anywhere in particular I could help out with.

[xnox]

I think we should start supporting XDG_CONFIG_HOME, probably the same way that git started to support XDG_CONFIG_HOME

 write to $XDG_CONFIG_HOME/git/config file if this file exists and the ~/.gitconfig file doesn’t.

[konvpalto]

Pushed 5150de5 to "next", please, test.

[muggenhor]

@konvpalto I've been using 5150de5 cherry-picked to v6.5.5 (8bc2f35) for about a week now.

First I ran with the cfg file still in ~/.offlineimaprc for two days, (starting last Sunday I believe). Then I moved it to ~/.config/offlineimap/config somewhere last Wednesday. No problems so far. And frankly the change is simple enough that I don't expect any problems with these two test scenarios.

[konvpalto]

Good, thanks for your testing. For the record, XDG support is the part of v6.5.6, https://github.com/OfflineIMAP/offlineimap/releases/tag/v6.5.6

[lahwaacz]

To complete the compliance to the XDG standard, the pidfile should be created in $XDG_RUNTIME_DIR instead of the (configurable) metadata dir.

[gioele]

@lahwaacz please fine another issue just for that problem.