Storing defaults for global flags in the config file (profiles)

[ncw]

Make a global config setting for rclone

In the rclone config file have configuration for "profiles". These are default settings for all the rclone flags.

So the flag "--delete-during" would be stored like this in the config in the default profile.

[:default]
delete_during = true

Rclone would provide the --profile flag to use a different profile and provide rclone config profile to edit the profile with a config profile editor wizard.

Backend flags should be settable in the profile also - basically any flag that rclone has should be settable in the profile (just like it is settable in the environment already).

Extension idea: could inherit from other profiles.

[harryqt]

I would also love to see this. But judging by the milestone shifting I'm guessing it's hard to implement?

[ncw]

would also love to see this. But judging by the milestone shifting I'm guessing it's hard to implement?

So much to do, so little time :-)

It isn't particularly tricky - fancy helping out?

[kevin-hanselman]

I'm looking to configure rclone to enable things like --progress globally by default. This issue looks more or less to fit the bill. Am I correct, or is there an alternative that I'm not grokking from the docs?

[ivandeex]

@kevin-hanselman
Currently you can use environment for that. Set default values in your ~/.profile. https://rclone.org/docs/#options-1

[ncw]

@kevin-hanselman

I'm looking to configure rclone to enable things like --progress globally by default. This issue looks more or less to fit the bill. Am I correct, or is there an alternative that I'm not grokking from the docs?

Yes this is the issue.

However as @ivandeex points out you could set an env var RCLONE_PROGRESS=1

[ivandeex]

We already have following config sources (ordered from low to high priority):
defaults, backend config, environment, cmdline.

This can be implemented as a new config source profile config.
I'm not sure about priority of the profile source. I think it should go between backend and environment above.

Inheritance of profile section child from parent can be expressed as

[:parent]
x = base_x
y = common_y
[:child:parent]
x = override_x
z = child_z

@albertony @ncw thoughts?

[cerealXDD]

Is this feature implemented yet? I want to add flags to all rclone commands by default without the shell.

[Tatsh]

Using alias for now.

alias rclone='rclone --progress --bwlimit 4.2M --transfers=1'

[dalbani]

An added benefit of supporting those flags in the config file, would be for situations where rclone is used via the API and not run through the CLI.
(Shameless plug: for example what I'm working on for SeaweedFS: seaweedfs/seaweedfs#4402.)

[ncw]

@dalbani wrote:

An added benefit of supporting those flags in the config file, would be for situations where rclone is used via the API and not run through the CLI.

I guess you mean the internal Go API here?

Would you like to work on this PR?

(Shameless plug: for example what I'm working on for SeaweedFS: seaweedfs/seaweedfs#4402.)

That is a very nice bit of work! I'll just note that we don't particularly guarantee stability of the internal rclone interfaces. I try not to break them though :-)

[nielash]

@ncw I took a stab at implementing this. It's still a WIP but if you have a moment, it would be very helpful if you could take a peek and let me know if I'm on the right track: nielash@60b9568

Features:

• save a profile while running any other command by adding --save-profile NAME
• use one or more profiles with a command by adding --use-profile NAME1,NAME2,NAME3...
• --profile-include-args to include the command's args in addition to flags (for example, two long paths that you sync regularly)
• Support for inheriting settings from multiple nested "parent" profiles
• Support for setting one "default" profile in the config, for use cases where users can't set --use-profile from command line flags or env vars
• new dedicated setprofile command
• new dedicated profiles wizard UI in rclone config profile

There are a number of (passing!) tests here that show the intended use:
https://github.com/nielash/rclone/blob/profiles-WIP/fs/config/configprofiles/profiles_test.go

[ncw]

@nielash

I took a stab at implementing this. It's still a WIP but if you have a moment, it would be very helpful if you could take a peek and let me know if I'm on the right track: nielash@60b9568

Very nice list of features :-) You've done a very thorough job.

This is going to need lots of docs!

I did also have the idea that we could introduce global flags to set when using a backend. So when you do

[myremote]
type = s3
whatever = blah
--inplace = true

This would set the --inplace flag which would be rather like using --profile myremote if you see what I mean.

So if we could arrange for the profile settings and the flag settings not to clash then we can enable this if we want.

Looking at the code you are using type = profile to tell the profiles apart from backends which seems sensible.

• save a profile while running any other command by adding --save-profile NAME

Very nice feature!

• use one or more profiles with a command by adding --use-profile NAME1,NAME2,NAME3...

This should probably just be --profile (eg the aws command).

• --profile-include-args to include the command's args in addition to flags (for example, two long paths that you sync regularly)

This should probably default to true - if you put args in the profile then you'd expect to use them.

• Support for inheriting settings from multiple nested "parent" profiles

How do you do that - I missed that in the code.

This could potentially be the mechanism for including a profile in a backend definition rather than including the flags directly.

• Support for setting one "default" profile in the config, for use cases where users can't set --use-profile from command line flags or env vars
• new dedicated setprofile command
• new dedicated profiles wizard UI in rclone config profile

All good!

There are a number of (passing!) tests here that show the intended use: https://github.com/nielash/rclone/blob/profiles-WIP/fs/config/configprofiles/profiles_test.go

Nice!

I think what this needs next is

• docs - this needs some really clear docs on how to use profiles, how they work, how they appear in the config file, how to edit them etc. This should be a section in docs.md and what we can do is run that by the users on the forum along with a link to a beta to get some feedback.
• can you try to break the commit up into smaller chunks for easier review please :-)
• we probably should have some API to do stuff with profiles, though maybe the existing config api will work

Very nice work :-)

[nielash]

Very helpful feedback -- thank you!

• Support for inheriting settings from multiple nested "parent" profiles

How do you do that - I missed that in the code.

In the config they are saved as a comma-separated list under the parent_profiles key, and then the p.Use method calls p.useParents, which eventually calls p.Use on each of those parents, which means doing the same for all of their parents, if any (grandparents), recursively until it runs out of ancestors to "use" 😆

The priority of conflicting settings is resolved in an order of lowest to highest, with parents all lower than child. (In other words, think of parents as a set of defaults that are inherited but can be overridden at will.)

This test shows it in action.

I did also have the idea that we could introduce global flags to set when using a backend.

I like this idea a lot. It is sort of already possible with the --profile-include-args implementation in the current draft. In other words, if you have this config:

[myremote]
type = s3
whatever = blah

[s3]
type = profile
inplace = true
arg_2 = myremote:

then rclone copy some:path --use-profile s3 should be equivalent to rclone copy some:path myremote: --inplace
(arg_2 as presumably the dest is what you care about with --inplace)

But this does have some drawbacks -- you'd need to know in advance which arg number it would be, and it is also not very flexible if you want to use some other path at that remote.

Tying global flags to remotes is a really interesting idea and I agree it would be useful. One question I have though is how conflicting flags would be handled in commands that use more than one remote. For example, suppose I sync two remotes, one of which has --checkers=8, the other of which has --checkers=16. How many checkers would be used for the sync? Another question to ponder: what if the src specifies dest-related flags like --inplace but the dest is silent on that flag -- is it used or ignored?

Another possible implementation to consider: what if an Fs remote could take an optional profile parameter to call p.Use maybe during NewFs? Then the profiles code doesn't have to be concerned with going into other config sections and parsing them directly -- it is more of a pointer to where the settings are defined. Maybe that is what you were getting at with the parent profiles idea.

I'll give this more thought, as I agree it would be quite handy! (A use that comes up for me a lot is wanting to use -c with backends that support checksums but omit it otherwise, to avoid the fallback to --size-only. It would be nice to be able to just put that -c in the config!)

This is going to need lots of docs!

Absolutely! That's a given.

This should probably just be --profile (eg the aws command).

I agree. I'll change it in the next draft.

--profile-include-args to include the command's args in addition to flags (for example, two long paths that you sync regularly)

This should probably default to true - if you put args in the profile then you'd expect to use them.

My slight hesitation about this is that I'm not sure if that assumption also holds true when saving a profile. For example, if I do:

rclone sync /some/path drive:somepath -MvP --create-empty-src-dirs --save-profile sync

would you expect that to save the paths too? Or just -MvP --create-empty-src-dirs? If we reversed the default, would you have to do --profile-no-args to save only the flags?

[HorseLuke]

@nielash Does it support stringArray in your implements? Some config flags can be passed multiple, eg --exclude, --ca-cert. Example: https://forum.rclone.org/t/rclone-exclude-not-working-for-multiple-rules/35200

[nielash]

@nielash Does it support stringArray in your implements? Some config flags can be passed multiple, eg --exclude, --ca-cert.

Very good point! Yes, it should be supported, however it would be merged into one key in the config. For example:

--exclude file1 --exclude file2

would become:

exclude = [file1,file2]

The reason is because if you did this:

exclude = file1
exclude = file2

it would be invalid INI (can't have duplicate keys in the same section)