Struggling to get command to work

[DarrenPIngram]

This is not a bug, but probably a bug in the head of the user.

I am trying to get Resticprofile to run. I can run Restic itself, using the following (I note the "transfers" bit has no actual benefit, but hadn't changed it pending a move to Resticprofile. (system is MacOS 10.15.7, installed via your "brew" instructions.

This is how (so far) I have run Restic in various tests.

restic -o rclone.program="/usr/local/bin/rclone" \
  -o rclone.args="serve restic --stdio --transfers 10 --verbose" \
  -r rclone:DR:restic/Appdata backup /Volumes/Appdata

I have configured this in Resticprofiles (DR is the Dropbox mount, configured through rclone, e.g. restic -r rclone:foo:bar init

di@MacPro .restic % resticprofile profiles
2021/10/27 13:41:28 using configuration file: profiles.conf

Profiles available (name, sections, description):
  appdata:   (n/a)  
  default:   (env)  
  no-cache:  (n/a)  

I can see the data by entering rclone ls DR:restic/Appdata

But if I launch Resticprofiles it brings up an error, and despite looking extensively at your notes I am struggling to see what I have done wrong.

I am trying to launch Resticprofiles by resticprofile -n appdata backup

Could someone look at it? Also if any other errors are noticed I welcome being corrected, as I continue my journey. The objective of this job is to backup /Volumes/Appdata to DR:restic/Appdata each time it is run. I will worry about scheduling later.

In /Users/di/.restic I have the following files

-rw-r--r--  1 di  staff    64 Oct 27 11:21 exclude.txt
-rw-r--r--  1 di  staff    14 Oct 27 11:22 password.txt
-rw-r--r--  1 di  staff  2394 Oct 27 13:51 profiles.conf

profiles.conf : pastebin https://pastebin.com/6QtexdF0

The objective would be to have different backup locations (DR=Dropbox, GD=Google Drive (both via rclone) with different volumes getting different repositories, e.g. DR:restic/Vol2, DR:restic/Vol3 and each Vol would get its own section in square brackets as I have done with Appdata

Thank you in advance for any guidance!

[jkellerer]

Hi @DarrenPIngram , resticprofile implements configuration profiles for restic but it has no extra logic e.g. to know that it must use rclone for a certain repo type. If you need rclone it must be added to the profile as "option" arguments. You can always check the effective commands that are generated by adding --dry-run and compare to the commands you would normally use.

E.g. to add rclone settings try the following in your profile:

repository = "rclone:DR:restic/Appdata"
option = [
    "rclone.program='/usr/local/bin/rclone'",
    "rclone.args='serve restic --stdio --transfers 10 --verbose'"
]

This should then have the following effect:

➜ resticprofile --dry-run --name appdata backup
2021/10/27 13:21:06 using configuration file: profiles.conf
2021/10/27 13:21:06 profile 'appdata': initializing repository (if not existing)
2021/10/27 13:21:06 dry-run: /usr/local/bin/restic init --option rclone.program='/usr/local/bin/rclone' --option "rclone.args='serve restic --stdio --transfers 10 --verbose'" --password-file password.txt --repo rclone:DR:restic/Appdata
2021/10/27 13:21:06 profile 'appdata': starting 'backup'
2021/10/27 13:21:06 dry-run: /usr/local/bin/restic backup --option rclone.program='/usr/local/bin/rclone' --option "rclone.args='serve restic --stdio --transfers 10 --verbose'" --password-file password.txt --repo rclone:DR:restic/Appdata /Volumes/Appdata
2021/10/27 13:21:06 profile 'appdata': finished 'backup'

[DarrenPIngram]

Thank you. I appreciate the very fast response. May I suggest a note in the configuration instructions explicitly referencing Rclone may help other people who, like me, maybe failed to appreciate everything and went down the rabbit hole wrongly?

I tried the dry-run and it seemed to be correct after adding the extra lines you suggested. I took away the --transfers 10 bit as that would have not been implemented going forward.

di@MacPro .restic % resticprofile --dry-run --name appdata backup
2021/10/27 14:41:17 using configuration file: profiles.conf
2021/10/27 14:41:17 profile 'appdata': starting 'backup'
2021/10/27 14:41:17 dry-run: /usr/local/bin/restic backup --exclude-caches --exclude-file exclude.txt --option rclone.program=\'/usr/local/bin/rclone\' --option rclone.args=\'serve\ restic\ --stdio\ --verbose\' --password-file password.txt --repo rclone:DR:restic/Appdata --tag Appdata --verbose /Volumes/Appdata
2021/10/27 14:41:17 profile 'appdata': finished 'backup'

However, running it "live" brought up a further error:

di@MacPro .restic % resticprofile --name appdata backup          
2021/10/27 14:43:13 using configuration file: profiles.conf
2021/10/27 14:43:13 profile 'appdata': starting 'backup'
open repository
rclone: Error: unknown command "serve restic --stdio --verbose" for "rclone"
rclone: Run 'rclone --help' for usage.
rclone: You could use 'rclone selfupdate' to get latest features.
rclone: 
rclone: 2021/10/27 14:43:14 Fatal error: unknown command "serve restic --stdio --verbose" for "rclone"
Fatal: unable to open repo at rclone:DR:restic/Appdata: error talking HTTP to rclone: Get "http://localhost/file-5577006791947779410": unexpected EOF
2021/10/27 14:43:14 backup on profile 'appdata': exit status 1
di@MacPro .restic % 

I have not needed to (knowingly) configure a Restic server for my previous backup commands to operate and I am already skating a bit of out of my depth with the Restic/Rclone knowledge. Looking at https://rclone.org/commands/rclone_serve_restic/ suggests that you only override if you have anything other than the 
```defaults, and I've not knowingly set anything.

I have double-checked and you can see the Rclone mount of DR:restic/Appdata:
di@MacPro .restic % rclone lsd DR:restic/Appdata  

di@MacPro .restic % rclone lsd DR:restic/Appdata
-1 2021-10-27 14:50:23 -1 data
-1 2021-10-27 14:50:23 -1 index
-1 2021-10-27 14:50:23 -1 keys
-1 2021-10-27 14:50:23 -1 locks
-1 2021-10-27 14:50:23 -1 snapshots

May I trouble you for a little more hand-holding?

[DarrenPIngram]

I will add I have not knowingly had anything stored locally, other than the cache files for rclone/restic that take a default location so far.

[jkellerer]

Fully agree that the documentation should provide an example with rclone as it is not uncommon.

This looks like an encoding bug in resticprofile: --option rclone.args=\'serve\ restic\ --stdio\ --verbose\' is likely passed literally to restic and this will then use it as a single argument to rclone instead of separate commands. Needs testing. @creativeprojects do you have an idea for a quick fix?

[DarrenPIngram]

Thank you. I am "happy" at least that the problem now is probably not of my own making. I look forward to hearing back, hopefully with a nice interim resolution. I am quite excited by this project as I expect it is going to make my life a lot, lot easier than what I had expected it to be, from looking at various elements of scripting Restic in general. ‌

…

On Wed, 27 Oct 2021 at 15:00, jkellerer ***@***.***> wrote: Fully agree that the documentation should provide an example with rclone as it is not uncommon. This looks like an encoding bug in resticprofile: --option rclone.args=\'serve\ restic\ --stdio\ --verbose\' is likely passed literally to restic and this will then use it as a single argument to rclone instead of separate commands. Needs testing. @creativeprojects <https://github.com/creativeprojects> do you have an idea for a quick fix? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#82 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABV5MAEREEN5LYMLTH7V67LUI7ZXHANCNFSM5G2BPUMQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[creativeprojects]

May I suggest a note in the configuration instructions explicitly referencing Rclone may help other people who, like me, maybe failed to appreciate everything and went down the rabbit hole wrongly?

I will definitely do that.

I just gave it a try, and you don't need to use single quotes. It should work if you change your config as such:

repository = "rclone:DR:restic/Appdata"
option = [
    "rclone.program=/usr/local/bin/rclone",
    "rclone.args=serve restic --stdio --transfers 10 --verbose"
]

...and I'll try to find a fix to make it work with single quotes 😄

[DarrenPIngram]

Perfect, that worked. Thank you. I do have some other questions and no doubt more in the future, is it OK to post as new "issues" (even though they are not necessarily bugs), or do you have a different preferred venue? One feature request, unless it is always possible and I've not figured it out, is the ability to run the profile with another job (in another tab) whilst one backup is running (e.g. to two or more separate backup jobs). I'm not yet in a position to put jobs in to a schedule, or run as a "group" either. ‌

…

On Wed, 27 Oct 2021 at 21:35, Fred ***@***.***> wrote: May I suggest a note in the configuration instructions explicitly referencing Rclone may help other people who, like me, maybe failed to appreciate everything and went down the rabbit hole wrongly? I will definitely do that. I just gave it a try, and you don't need to use single quotes. It should work if you change your config as such: repository = "rclone:DR:restic/Appdata" option = [ "rclone.program=/usr/local/bin/rclone", "rclone.args=serve restic --stdio --transfers 10 --verbose" ] ...and I'll try to find a fix to make it work with single quotes 😄 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#82 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABV5MADC4SGN5F5VLCMF3NDUJBIALANCNFSM5G2BPUMQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[creativeprojects]

I do have some other questions and no doubt more in the future, is it OK to
post as new "issues" (even though they are not necessarily bugs), or do you
have a different preferred venue?

You can also use the Discussions for that matter

One feature request, unless it is always possible and I've not figured it
out, is the ability to run the profile with another job (in another tab)
whilst one backup is running (e.g. to two or more separate backup jobs).

If your backup target is different you can already do that.
If the backup target is the same I would probably not want to do that.