QEP 82: Unified Application Profiles

[NathanW2]

QGIS Enhancement 82: Unified Application Configuration

Date 2016/11/12
Author Nathan Woodrow @NathanW2
Contact woodrow.nathan@gmail.com
maintainer @NathanW2
Version QGIS 3.0

Summary

This QEP is to add and expand Unified Application Configuration support into core QGIS. The main idea is to bundle everything that is normally in .qgis2 and settings (registry on
windows, config files on OS X/*nix, etc) into a single folder in the users platform based settings folder. This would allow the creation of saved config folder, you can think of them like profiles in Chrome/Firebreak, etc

The config folder would be a single folder with all .qgis2 and settings files which currently the result of doing the following in QGIS 2:

qgis --configpath folder\qgis

As saved configs contain isolated settings and plugins they can be great for different workflows, demos, users of the same machine, or testing settings, etc.

Proposal includes removing the use of current .qgis2 and settings paths (registry on Windows) in QGIS 3.x in favour of new profiles folder setup.

I think this method of storing settings and plugins, etc, could remove a lot of current confusion in how QGIS stores settings, as well as increase the flexibility in enterprise style deployments of QGIS.

Backups of user profiles also because very easy. A simple copy the profile folder to make a backup.

Proposed Solution

This proposal is to add new option --profile, as well as UI related to creating and switching saved configs.

All configs would live in a single top level folder in the platform dependent location e.g appdata on windows, etc, A link the about screen would also have the option to locate the current profile folder in the OS file manager.

An example of profile layout:

%APPDATA%
 - \QGIS
   - \configs
      - .default
      - \config1
      - \config2
      - \config3

By default, QGIS would run using the default config which is found in .default, e.g profile1

Part of this QEP is also loading machine-wide settings that are installed in the QGIS install folder. These should be loaded and then joined with the users saved config settings. Settings will follow the same format as the a users folder setup however there is only one.

Affected Files.

main() of QGIS would be altered to load configs from the settings folder.

A new class will also be introduced called QgsSettings which hides the implementation of config loading away from the code base. QgsSettings will be used in place of QSettings though out the core code base.

QgsSettings will have an API similar to QSettings in order to be a drop in replacement. The current implementation is to just be a wrapper for QSettings however in the future this could hold other logic for finding settings.

Performance Implications

None.

Further Considerations/Improvements

Cloud config

By hiding any implementation details inside QgsSettings we can also add loading config from a database and webserver in the future.

Webserver could expose endpoints to download/upload zip files of the user's config folder into a hosted environment. This is out of the scope of this QEP, however, this can be used as a building block for that function in future.

Backups

Built in backup process to allow rollbacks of settings. As the settings file is simply text files each change and is backed up and compressed for later rollback.

Backwards Compatibility

If QGIS 3.0 will inherit the settings of QGIS 2.x would migrate any existing settings and plugins into the new profile folder layout.

Issue Tracking ID(s)

(optional)

[pcav]

Agreed. Very good improvement. Thanks Nathan.

[nyalldawson]

All profiles would live in a single top level folder, maybe in ~, %HOME% to
make it easy for the admins and users to find if needed. A link the about
screen would also have the option to locate the current profile folder.

I think we should respect the standard folder locations here. So Appdata\Roaming (or local?) for windows, and ~/.config/qgis for linux. Not sure what the OSX standard is.

We've been abusing people's home folders with the .qgis2 folder for a long time, it'd be good to start following the standard now that we've got the chance.

See also http://hub.qgis.org/issues/11926

At the same time we could also should split up config and cache folders, so users aren't unnecessarily backing up temporary files when they are saving their qgis settings.

[nyalldawson]

Just so I understand this fully - this means no more registry use on windows?

[NathanW2]

Yes, no more registry on Windows. A single uniform setup for all platforms. Happy to follow the standard folder layout.

…

On Fri, Nov 25, 2016 at 9:53 AM, Nyall Dawson ***@***.***> wrote: Just so I understand this fully - this means no more registry use on windows? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#82 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAXS3OGe43Ziv35w8LXgqEsT1mzwBBdMks5rBiOGgaJpZM4KwZyI> .

[NathanW2]

If we provide a way to jump to the folder from inside QGIS it doesn't really matter the location it's saved. So happy to match what the standard is for each platform.

[nyalldawson]

When would someone need to jump to this folder? For backups or something?

[NathanW2]

Yeah, just generally I don't think it's a good idea to not have a quick way of getting there. Of course, people really shouldn't but just in case. We could also provide UI for exporting/importing, etc

…

On Fri, Nov 25, 2016 at 10:23 AM, Nyall Dawson ***@***.***> wrote: When would someone need to jump to this folder? For backups or something? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#82 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAXS3CeD9i8ZencnnQsWDu-RKN8417xWks5rBiqYgaJpZM4KwZyI> .

[pcav]

+1 for getting rid of win registry
+1 for a standard location
unsure about designing our way to manage the config folder: what other major sw do? Can't we leave this to the OS file manager?

[NathanW2]

Yes, I just mean have a way to open that folder e.g this works cross platform QDesktopServices::openUrl(QUrl::fromLocalFile("path to folder")) that will open that folder using the OS manager.

…

On Fri, Nov 25, 2016 at 4:56 PM, Paolo Cavallini ***@***.***> wrote: +1 for getting rid of win registry +1 for a standard location unsure about designing our way to manage the config folder: what other major sw do? Can't we leave this to the OS file manager? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#82 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAXS3Dk5XUnc5QJnZiYYoY_jfXCKrIHbks5rBoaAgaJpZM4KwZyI> .

[ninsbl]

Yes, indeed very nice to get rid of registry use, which several others already wished for on the mailinglists!

Not sure if that can be within the scope of this QEP, but in an (our) organizational setting, we have some settings (e.g. connections to OWS) which are frequently used by almost all users. These were put into a default setting, which is the copied into the user-homer. Users then can adjust the settings and add more OWS and so on. However, OWS-url get updated now and then and the question arises how to update the URL in all user settings without overwriting changes/adjustments users made to their QGIS.ini.

What I would like to have would be the flexibility, that users can change the settings as they like, but that "central" settings can be updated by an administrator (and not every user has to do that himself). A kind of versioning of the QGIS.ini. Splitting up the Settings into different sections seems to be helpful in that direction.

But maybe that is something for a Python-Macro...

[haubourg]

Hi, this is a very welcome change!
+1 for standard location by default, + still a configpath option to move the profile elsewhere.

BTW, if we throw some light on profiles, I remember that some ressources can't be centralized easily by adding extra paths to qgis config.
We currently can add additionnal paths to SVG folders, plugins folders, but we miss that for symbols (markers, color ramps aka palettes.), and processing models (and also themes, and project templates). Ressource sharing is great for individual users but not adapted for centralized systems. Currently, admins have to deploy complex ini file management strategies for those parts. Adding to extra paths options in QGIS would make life easier for many of us :). Sorry if I'm off topic.

[rduivenvoorde]

Yes, I also like this idea's very much!!

Note that it is (at least on commandline in linux) already very easy using --configpath (which then takes BOTH settings and config). This also makes it easy to start with a clean slate or use different configs. If I remember correct @wonder-sk and @blazek did this for the configuration-work for schools.

I have QGIS config's for several client-projects within the client-project folder and start QGIS from command line: qgis --configpath /home/richard/zuidt/16/client_project/qgisfolder
In that way I also keep client-specific plugins/data/proxysettings etc etc together.

Firefox has the option to start a 'profile manager' using the -P option:

Off course for windows/mac it is probably better to have a second small binary for that?

Do you think we still would need 'application'-settings? Or a concept in which you can 'save' options either in 'current' config, or in the 'global'-config? I'm thinking about for example: the theme to use (I would never use the 'Vista'-theme anymore on windows for example), or the place where you put all your icons? We could have a concept in which QGIS first loads application settings and then overrides some of those via the config settings?

Another idea: I find the starting up of QGIS sometimes slow. So IF we put a 'config'-chooser in the startup process, some core part of QGIS can be started very quick (in parallel when the 'choose config' dialog is shown)... and then only starts what is nessecary. At least making the impression that we start up faster would be nice I think.

Another idea: IF we have this minimal config choose already, would it be an option (in case of a debug build) to stream the debug output to that window too? In that case it would make it easier for windows/mac users to see what is happening when something goes wrong. I always do debug builds, and when something goes wrong it is the first place I look into....

[elpaso]

+1 from me, I cannot see any negative implications, and I see only clear advantages.

[ninsbl]

The --configpath option is available on Windows too.

[NathanW2]

Hmm how did I not see the --configpath option already. Good to know. I will update the QEP.

Most of this is still valid as I guess it's more about making this the default setup and I would still add a --profile option which can just take a name of a profile to load.

--configpath can be used for non profile based setups e.g save settings on a network drive or something.

[rduivenvoorde]

@NathanW2 mmm, please do not overload the user with concepts: we already had 'config' and 'settings' and now we have 'profile' too?

[NathanW2]

@rduivenvoorde profile is a higher level concept a named config if you will, most users will never load QGIS on the command line.

If it's better maybe we change all the naming from profile to config and use that? I'm not fixed on the profile naming so happy to roll with just calling it config.

[NathanW2]

@rduivenvoorde A config loader is a good idea, I will add that.

On the slow starting, I suspect this is due to the large start method in code and also in the plugins. A different topic that we will need to dig into.

[rduivenvoorde]

@NathanW2 yeah, I'm ok with whatever naming, it's just that I do not want to overload people with (our) abstract ideas as what they want is just "have all my QGIS configurations" in one place.
Do you also think there should be two levels of configurations (global and profile)?

[pcav]

Agreed, having different user profiles and the possibility of switching is a long standing idea, it would be great to see it implemented.
Thanks.

[Frederikssund]

Hi Nathan - A very welcome change!!

Just a couple of additions:

It would nice if it was possible to use some "variable substitution" mechanism inside the setup files.

Use Case: When I have to prepare a rollout of Qgis to a "enterprise size" number of users (250), I'll do the following:

1. Install a normal Qgis on a "clean" computer.
2. Change the startup to use --configpath qualifierer, which points to a non-existing directory inside the Qgis program directory
3. Prepare my installation by adding plugins, changing setup options and so on. This will create the abovementioned directory with setup information for Qgis
4. At last distribute the new Qgis installation essentially by copying the Qgis program directory to the end users computer.
5. The first time startup of the new Qgis with a slightly modified qgis startup file triggers a copy of the user directory prepared by me to a final location on the users computers.
   And now comes the problem.... The qgis.ini setup file contains a boatload af location references that points to the original location of the user directory. This is taken care of by running a sed based "search and replace" function that changes the original location references to the final location of the QGis userdirectory in the qgis.ini file. This "search and replace" sed command is very shaky (and butt-ugly to look at). It would be much simpler to use some kind a variable mechanism inside the ini file and simply change the value of the variable (ex. the location of the Qgis program directory and the qgis user-directory) when it's necessary.

In case the above is total gibberish, I have a detailed description of the process at https://github.com/Frederikssund/Alternativ-QGIS-installation

Suggestion no. 2: Put the microsoft support dll's ( msvcp*.dll and msvcr*.dll) in the qgis bin directory. That would make the Qgis program file directory "self contained" with no external file dependencies

Suggestion no 3: Add a user accesible "profile switcher", so it's possible to change profile dynamically during a session (A restart af Qgis it acceptable ;-)

Suggestion no 4: Loading the plugins sequentially (and not using threads) slows Qgis a lot during startup.
A nice easy method to use threads during plugin initialization would help a lot !

Regards Bo Victor

[ninsbl]

Use Case: When I have to prepare a rollout of Qgis to a "enterprise size" number of users
Pretty similar here compared to what @Frederikssund does and experiences.

A "profile switcher" would be great, because this is currently quite cumbersome on Windows esp. if one does not have admin rights...

For relative path / variable support there is also: https://hub.qgis.org/issues/12623

However, I am afraid discussion is drifting away from the original intention of the QEP...

[SrNetoChan]

Very good idea. Thanks for that.

The only negative implication that I can think of is multiplying the same plugin(s) in several different profiles. But not sure if that is really an issue.

[pcav]

@SrNetoChan I do not think the extra Mb wasted for multiple copies of the plugin are a problem. In fact, this would allow for testing new version, having different sets installed, etc.

[Frederikssund]

The "profile switcher" idea is explored by Boundless: "http://boundlessgeo.github.io/qgis-plugins-documentation/profiles/plugin.html". However it would be nice if its baked into Qgis proper instead of being a plugin.

[NathanW2]

I will update this QEP tonight with more notes on the final design

[elpaso]

Here is a first C++ quick and dirty prototype bound to a single setting as a proof of concept.
planetfederal/QGIS@042abdf
Note: I developed this as a proof of concept before our yesterday's conversation, I'm posting the link just in case we can re-use something.

[NathanW2]

@elpaso that looks good to me. mind if I just yank that into my code?

[elpaso]

@NathanW2 not at all, I can port the python tests to C++ if you like, or write the bindings and use Python for the tests too. Currently missing from the implementation are the array read/write methods though.

[NathanW2]

@dakcarto @elpaso how do you feel about the machine settings using the same format as what user profiles do? e.g on windows it might look like this:

This makes it easy to "deploy" machine settings by just building it up as a local config and then putting it in the install folder?

Does that make sense?

We might have do some more advanced merging logic for some of the other things but I think it should work ok.

[NathanW2]

@rduivenvoorde I have update the QEP to now call it "configs" and not profiles to avoid any confusion with computer profiles and logins.

[elpaso]

@NathanW2 looks good to me. We need an easy way to create the default (machine) settings and I cannot think of an easiest one than copying from and existing configuration.

[dakcarto]

@NathanW2 wrote:

how do you feel about the machine settings using the same format as what user profiles do?

That seems fine to me, especially if inheritance (in @elpaso's prototype) is to work properly.

Maybe there could be checkboxes added to the Options -> Advanced -> Advanced Settings Editor, then one could just select what to export to a custom settings file in INI format (i.e. the QSettings-style INI that is, plus anything needed for custom configs in this QEP). Might be a simple way to generate the default (install-relative) settings file.

[dakcarto]

Or, do you mean the install-relative defaults config would actually be a directory, instead of a single config file?

[NathanW2]

Yeah it would just be a folder with the same layout as the users config but in the install folder.

…

On Tue, Jan 31, 2017 at 9:32 AM, Larry Shaffer ***@***.***> wrote: Or, do you mean the install-relative defaults config would actually be a directory, instead of a file? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#82 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAXS3M0DmIVmJP-R6HQwFnXYkrZgGZz4ks5rXnMcgaJpZM4KwZyI> .

[dakcarto]

Ok. So if one were to just want to set defaults, add [QGIS]/QGIS3 to the QgsApplication::pkgDataPath()/<some standard name for defaults>/ directory? Or, some variation on that?

[NathanW2]

Yeah

…

On Tue, Jan 31, 2017 at 9:39 AM, Larry Shaffer ***@***.***> wrote: Ok. So if one were to just want to set defaults, add [QGIS]/QGIS3 to the QgsApplication::pkgDataPath()/<some standard name for defaults>/ directory? Or, some variation on that? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#82 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAXS3Byh7wH9OJqkcdEX5Agkkx8N0nieks5rXnTFgaJpZM4KwZyI> .

[dakcarto]

Seems like the install-relative defaults wouldn't necessarily be something you would want copied over to a user config directory, if the referenced directory is also the template for such a user config.

Or, maybe you do, as a starting point for editing, since, in user space, those settings are just user overrides, not defaults?

[NathanW2]

No they aren't copied over. I'm just saying we use the same folder layout in the install folder as we do for the user configs. Or we can just have it load a .ini file if that is easier. I was just trying to think if we wanted to be able to deploy other non settings based stuff, like symbols etc.

…

On Tue, Jan 31, 2017 at 9:45 AM, Larry Shaffer ***@***.***> wrote: Seems like the install-relative defaults wouldn't necessarily be something you would want copied over to a user config directory, if the referenced directory is also the template for such a user config. Or, maybe you do, as a starting point for editing, since, in user space, those settings are just user overrides, not defaults? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#82 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAXS3GUOj85AeDVGbWlFaCAnwsaMu2Mtks5rXnYpgaJpZM4KwZyI> .

[dakcarto]

Ah, ok, good point. Yeah, seems like a good idea to me.

[elpaso]

@NathanW2 @dakcarto I've updated the dev branch with a full implementation with bindings and tests for the QgsSettings class: https://github.com/boundlessgeo/QGIS/tree/qgssettings-prototype
https://github.com/boundlessgeo/QGIS/blob/qgssettings-prototype/src/core/qgssettings.h

doxigen comments are still missing and I'd probably add some more shortcuts.

[haubourg]

I just had a look of current master's status concerning profiles management.
--configpath has no more effect and is replaced by the profiles-path and profile-name.

Shouldn't we raise a message if a qgis session is launched with deprecated options ?

[NathanW2]

Yep that is a good idea.

…

On Fri, Sep 1, 2017 at 12:32 AM, Régis Haubourg ***@***.***> wrote: I just had a look of current master's status concerning profiles management. --configpath has no more effect and is replaced by the profiles-path and profile-name. Shouldn't we raise a message if a qgis session is launched with deprecated options ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#82 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAXS3K6RM-KXxIZ0_zoR4r4MHOeajkqwks5sdsQAgaJpZM4KwZyI> .