Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Server-Side move is not possible with Chunker-Crypt-Remote #5939

Open
jdr85 opened this issue Jan 23, 2022 · 2 comments
Open

Server-Side move is not possible with Chunker-Crypt-Remote #5939

jdr85 opened this issue Jan 23, 2022 · 2 comments
Assignees
@ivandeex
Labels
bug Remote: Chunker
Milestone
Known Problem

Comments

@jdr85
Copy link

jdr85 commented Jan 23, 2022

The associated forum post URL from https://forum.rclone.org

https://forum.rclone.org/t/deleted-files-re-uploaded-when-using-backup-dir-with-chunker-and-crypt/28757/3

What is the problem you are having with rclone?

Server-Side move is not possible with Chunker-Crypt-Remote. I first noticed this when I found that all files I wanted to move to backup-dir were re-uploaded instead of doing a server-side move. I did some further tests and found that server-side move generally

  • works with (unencrypted) remote
  • works with crypt-remote
  • does not work with chunker-crypt-remote

What is your rclone version (output from rclone version)

rclone v1.57.0
- os/version: ubuntu 20.04 (64 bit)
- os/kernel: 5.4.0-96-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17.2
- go/linking: static
- go/tags: none

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

  • Ubuntu Server 20.04 on one machine
  • Debian 11 on the other machine

Which cloud storage system are you using? (e.g. Google Drive)

Jottacloud, Magentacloud --> seems to be cloud storage independent.

The relevant parts of my rclone.conf (although not requested here)

[Magentacloud]
type = webdav
url = https://magentacloud.de/remote.php/webdav
vendor = nextcloud
user = xxxx
pass = xxxx

[Crypt-Mc]
type = crypt
remote = Magentacloud:/rclone-test
password = xxxx

[Chunker-Crypt-Mc]
type = chunker
remote = Crypt-Mc:
chunk_size = 32Mi
hash_type = none

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

rclone sync with move to backup-dir:

rclone -vv sync --backup-dir=Chunker-Crypt-Mc:/rclone-test_bu/ /daten/_jdr/rclone-test/ Chunker-Crypt-Mc:/rclone-test/

rclone move of one existing file:

rclone -vv move Chunker-Crypt-Mc:/rclone-test/debian-11.2.0-i386-netinst.iso Chunker-Crypt-Mc:/rclone-test2/

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

Log - sync to crypt-storage only (which works):

rclone -vv sync --backup-dir=Crypt-Mc:/rclone-test_bu/ /daten/_jdr/rclone-test/ Crypt-Mc:/rclone-test/
2022/01/22 19:02:17 DEBUG : Setting --config "/daten/_jdr/rclone.conf" from environment variable RCLONE_CONFIG="/daten/_jdr/rclone.conf"
2022/01/22 19:02:17 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "-vv" "sync" "--backup-dir=Crypt-Mc:/rclone-test_bu/" "/daten/_jdr/rclone-test/" "Crypt-Mc:/rclone-test/"]
2022/01/22 19:02:17 DEBUG : Creating backend with remote "/daten/_jdr/rclone-test/"
2022/01/22 19:02:17 DEBUG : Using config file from "/daten/_jdr/rclone.conf"
2022/01/22 19:02:17 DEBUG : Creating backend with remote "Crypt-Mc:/rclone-test/"
2022/01/22 19:02:17 DEBUG : Creating backend with remote "Magentacloud:/rclone-test/dp9klrdcs1jj9b75m205qui5j4"
2022/01/22 19:02:17 DEBUG : found headers:
2022/01/22 19:02:18 DEBUG : fs cache: renaming cache item "Magentacloud:/rclone-test/dp9klrdcs1jj9b75m205qui5j4" to be canonical "Magentacloud:rclone-test/dp9klrdcs1jj9b75m205qui5j4"
2022/01/22 19:02:18 DEBUG : fs cache: switching user supplied name "Magentacloud:/rclone-test/dp9klrdcs1jj9b75m205qui5j4" for canonical name "Magentacloud:rclone-test/dp9klrdcs1jj9b75m205qui5j4"
2022/01/22 19:02:18 DEBUG : Creating backend with remote "Crypt-Mc:/rclone-test_bu/"
2022/01/22 19:02:18 DEBUG : Creating backend with remote "Magentacloud:/rclone-test/66k9iph1rjsv2hi5hjpg9h0c84"
2022/01/22 19:02:18 DEBUG : found headers:
2022/01/22 19:02:20 DEBUG : fs cache: renaming cache item "Magentacloud:/rclone-test/66k9iph1rjsv2hi5hjpg9h0c84" to be canonical "Magentacloud:rclone-test/66k9iph1rjsv2hi5hjpg9h0c84"
2022/01/22 19:02:20 DEBUG : fs cache: switching user supplied name "Magentacloud:/rclone-test/66k9iph1rjsv2hi5hjpg9h0c84" for canonical name "Magentacloud:rclone-test/66k9iph1rjsv2hi5hjpg9h0c84"
2022/01/22 19:02:22 DEBUG : Encrypted drive 'Crypt-Mc:/rclone-test/': Waiting for checks to finish
2022/01/22 19:02:22 DEBUG : clonezilla-live-2.7.1-22-i686-pae.iso: Size and modification time the same (differ by -815.389049ms, within tolerance 1s)
2022/01/22 19:02:22 DEBUG : clonezilla-live-2.7.1-22-i686-pae.iso: Unchanged skipping
2022/01/22 19:02:22 DEBUG : Encrypted drive 'Crypt-Mc:/rclone-test/': Waiting for transfers to finish
2022/01/22 19:02:22 DEBUG : Waiting for deletions to finish
2022/01/22 19:02:44 INFO  : debian-11.2.0-i386-netinst.iso: Moved (server-side)
2022/01/22 19:02:44 INFO  : debian-11.2.0-i386-netinst.iso: Moved into backup dir
2022/01/22 19:02:44 INFO  : There was nothing to transfer
2022/01/22 19:02:44 INFO  :
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:                 3 / 3, 100%
Deleted:                1 (files), 0 (dirs)
Renamed:                1
Elapsed time:        27.2s

2022/01/22 19:02:44 DEBUG : 3 go routines active

Log - sync to chunker-crypt-storage (which does not work):

rclone -vv sync --backup-dir=Chunker-Crypt-Mc:/rclone-test_bu/ /daten/_jdr/rclone-test/ Chunker-Crypt-Mc:/rclone-test/
2022/01/22 19:11:23 DEBUG : Setting --config "/daten/_jdr/rclone.conf" from environment variable RCLONE_CONFIG="/daten/_jdr/rclone.conf"
2022/01/22 19:11:23 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "-vv" "sync" "--backup-dir=Chunker-Crypt-Mc:/rclone-test_bu/" "/daten/_jdr/rclone-test/" "Chunker-Crypt-Mc:/rclone-test/"]
2022/01/22 19:11:23 DEBUG : Creating backend with remote "/daten/_jdr/rclone-test/"
2022/01/22 19:11:23 DEBUG : Using config file from "/daten/_jdr/rclone.conf"
2022/01/22 19:11:23 DEBUG : Creating backend with remote "Chunker-Crypt-Mc:/rclone-test/"
2022/01/22 19:11:23 DEBUG : Creating backend with remote "Crypt-Mc:/rclone-test"
2022/01/22 19:11:23 DEBUG : Creating backend with remote "Magentacloud:/rclone-test/dp9klrdcs1jj9b75m205qui5j4"
2022/01/22 19:11:23 DEBUG : found headers:
2022/01/22 19:11:24 DEBUG : fs cache: renaming cache item "Magentacloud:/rclone-test/dp9klrdcs1jj9b75m205qui5j4" to be canonical "Magentacloud:rclone-test/dp9klrdcs1jj9b75m205qui5j4"
2022/01/22 19:11:24 DEBUG : fs cache: switching user supplied name "Magentacloud:/rclone-test/dp9klrdcs1jj9b75m205qui5j4" for canonical name "Magentacloud:rclone-test/dp9klrdcs1jj9b75m205qui5j4"
2022/01/22 19:11:24 DEBUG : Reset feature "ListR"
2022/01/22 19:11:24 DEBUG : Creating backend with remote "Chunker-Crypt-Mc:/rclone-test_bu/"
2022/01/22 19:11:24 DEBUG : Creating backend with remote "Crypt-Mc:/rclone-test_bu"
2022/01/22 19:11:24 DEBUG : Creating backend with remote "Magentacloud:/rclone-test/66k9iph1rjsv2hi5hjpg9h0c84"
2022/01/22 19:11:24 DEBUG : found headers:
2022/01/22 19:11:25 DEBUG : fs cache: renaming cache item "Magentacloud:/rclone-test/66k9iph1rjsv2hi5hjpg9h0c84" to be canonical "Magentacloud:rclone-test/66k9iph1rjsv2hi5hjpg9h0c84"
2022/01/22 19:11:25 DEBUG : fs cache: switching user supplied name "Magentacloud:/rclone-test/66k9iph1rjsv2hi5hjpg9h0c84" for canonical name "Magentacloud:rclone-test/66k9iph1rjsv2hi5hjpg9h0c84"
2022/01/22 19:11:25 DEBUG : Reset feature "ListR"
2022/01/22 19:11:26 DEBUG : Chunked 'Chunker-Crypt-Mc:/rclone-test/': Waiting for checks to finish
2022/01/22 19:11:26 DEBUG : clonezilla-live-2.7.1-22-i686-pae.iso: Size and modification time the same (differ by -815.389049ms, within tolerance 1s)
2022/01/22 19:11:26 DEBUG : clonezilla-live-2.7.1-22-i686-pae.iso: Unchanged skipping
2022/01/22 19:11:26 DEBUG : Chunked 'Chunker-Crypt-Mc:/rclone-test/': Waiting for transfers to finish
2022/01/22 19:11:26 DEBUG : Waiting for deletions to finish
2022/01/22 19:11:35 DEBUG : debian-11.2.0-i386-netinst.iso: Can't move - required hash not found
2022/01/22 19:11:35 DEBUG : debian-11.2.0-i386-netinst.iso: Can't move, switching to copy
2022/01/22 19:11:35 DEBUG : debian-11.2.0-i386-netinst.iso: Can't copy - required hash not found
2022/01/22 19:12:24 INFO  :
Transferred:      101.150 MiB / 470 MiB, 22%, 2.163 MiB/s, ETA 2m50s
Checks:                 1 / 2, 50%
Deleted:                1 (files), 0 (dirs)
Transferred:            0 / 1, 0%
Elapsed time:       1m1.3s
Checking:

Transferring:
 *                debian-11.2.0-i386-netinst.iso: 21% /470Mi, 2.209Mi/s, 2m46s
(Ctrl-C)

Log - move existing file to another dir (which does not work):

rclone -vv move Chunker-Crypt-Mc:/rclone-test/debian-11.2.0-i386-netinst.iso Chunker-Crypt-Mc:/rclone-test2/
2022/01/23 06:23:49 DEBUG : Setting --config "/daten/_jdr/rclone.conf" from environment variable RCLONE_CONFIG="/daten/_jdr/rclone.conf"
2022/01/23 06:23:49 DEBUG : rclone: Version "v1.57.0" starting with parameters ["rclone" "-vv" "move" "Chunker-Crypt-Mc:/rclone-test/debian-11.2.0-i386-netinst.iso" "Chunker-Crypt-Mc:/rclone-test2/"]
2022/01/23 06:23:49 DEBUG : Creating backend with remote "Chunker-Crypt-Mc:/rclone-test/debian-11.2.0-i386-netinst.iso"
2022/01/23 06:23:49 DEBUG : Using config file from "/daten/_jdr/rclone.conf"
2022/01/23 06:23:49 DEBUG : Creating backend with remote "Crypt-Mc:/rclone-test/debian-11.2.0-i386-netinst.iso"
2022/01/23 06:23:50 DEBUG : Creating backend with remote "Magentacloud:/rclone-test/dp9klrdcs1jj9b75m205qui5j4/ltitr2da9pi9v82se0j9bp538t9th273kohu28hsj0opnvce9h8g"
2022/01/23 06:23:50 DEBUG : found headers:
2022/01/23 06:23:50 DEBUG : fs cache: adding new entry for parent of "Magentacloud:/rclone-test/dp9klrdcs1jj9b75m205qui5j4/ltitr2da9pi9v82se0j9bp538t9th273kohu28hsj0opnvce9h8g", "Magentacloud:rclone-test/dp9klrdcs1jj9b75m205qui5j4"
2022/01/23 06:23:50 DEBUG : Reset feature "ListR"
2022/01/23 06:23:50 DEBUG : Creating backend with remote "Chunker-Crypt-Mc:/rclone-test2/"
2022/01/23 06:23:50 DEBUG : Creating backend with remote "Crypt-Mc:/rclone-test2"
2022/01/23 06:23:50 DEBUG : Creating backend with remote "Magentacloud:/rclone-test/pa6u8aehhjci6hdu6luhh10j50"
2022/01/23 06:23:50 DEBUG : found headers:
2022/01/23 06:23:50 DEBUG : fs cache: renaming cache item "Magentacloud:/rclone-test/pa6u8aehhjci6hdu6luhh10j50" to be canonical "Magentacloud:rclone-test/pa6u8aehhjci6hdu6luhh10j50"
2022/01/23 06:23:50 DEBUG : fs cache: switching user supplied name "Magentacloud:/rclone-test/pa6u8aehhjci6hdu6luhh10j50" for canonical name "Magentacloud:rclone-test/pa6u8aehhjci6hdu6luhh10j50"
2022/01/23 06:23:50 DEBUG : Reset feature "ListR"
2022/01/23 06:23:51 DEBUG : debian-11.2.0-i386-netinst.iso: Need to transfer - File not found at Destination
2022/01/23 06:23:51 DEBUG : debian-11.2.0-i386-netinst.iso: Can't move - required hash not found
2022/01/23 06:23:51 DEBUG : debian-11.2.0-i386-netinst.iso: Can't move, switching to copy
2022/01/23 06:23:51 DEBUG : debian-11.2.0-i386-netinst.iso: Can't copy - required hash not found
2022/01/23 06:24:50 INFO  :
Transferred:      251.812 MiB / 470 MiB, 54%, 4.799 MiB/s, ETA 45s
Checks:                 0 / 1, 0%
Transferred:            0 / 1, 0%
Elapsed time:       1m0.9s
Checking:

Transferring:
 *                debian-11.2.0-i386-netinst.iso: 53% /470Mi, 4.799Mi/s, 45s

2022/01/23 06:25:45 INFO  : debian-11.2.0-i386-netinst.iso.rclone_chunk.001_5ixyt8: Moved (server-side) to: debian-11.2.0-i386-netinst.iso.rclone_chunk.001
2022/01/23 06:25:46 INFO  : debian-11.2.0-i386-netinst.iso.rclone_chunk.002_5ixyt8: Moved (server-side) to: debian-11.2.0-i386-netinst.iso.rclone_chunk.002
2022/01/23 06:25:46 INFO  : debian-11.2.0-i386-netinst.iso.rclone_chunk.003_5ixyt8: Moved (server-side) to: debian-11.2.0-i386-netinst.iso.rclone_chunk.003
2022/01/23 06:25:47 INFO  : debian-11.2.0-i386-netinst.iso.rclone_chunk.004_5ixyt8: Moved (server-side) to: debian-11.2.0-i386-netinst.iso.rclone_chunk.004
2022/01/23 06:25:47 INFO  : debian-11.2.0-i386-netinst.iso.rclone_chunk.005_5ixyt8: Moved (server-side) to: debian-11.2.0-i386-netinst.iso.rclone_chunk.005
2022/01/23 06:25:48 INFO  : debian-11.2.0-i386-netinst.iso.rclone_chunk.006_5ixyt8: Moved (server-side) to: debian-11.2.0-i386-netinst.iso.rclone_chunk.006
2022/01/23 06:25:48 INFO  : debian-11.2.0-i386-netinst.iso.rclone_chunk.007_5ixyt8: Moved (server-side) to: debian-11.2.0-i386-netinst.iso.rclone_chunk.007
2022/01/23 06:25:49 INFO  : debian-11.2.0-i386-netinst.iso.rclone_chunk.008_5ixyt8: Moved (server-side) to: debian-11.2.0-i386-netinst.iso.rclone_chunk.008
2022/01/23 06:25:49 INFO  : debian-11.2.0-i386-netinst.iso.rclone_chunk.009_5ixyt8: Moved (server-side) to: debian-11.2.0-i386-netinst.iso.rclone_chunk.009
2022/01/23 06:25:50 INFO  : debian-11.2.0-i386-netinst.iso.rclone_chunk.010_5ixyt8: Moved (server-side) to: debian-11.2.0-i386-netinst.iso.rclone_chunk.010
2022/01/23 06:25:50 INFO  : debian-11.2.0-i386-netinst.iso.rclone_chunk.011_5ixyt8: Moved (server-side) to: debian-11.2.0-i386-netinst.iso.rclone_chunk.011
2022/01/23 06:25:50 INFO  :
Transferred:      470.115 MiB / 470 MiB, 100%, 2.980 MiB/s, ETA -
Checks:                11 / 13, 85%
Renamed:               11
Transferred:            0 / 1, 0%
Elapsed time:       2m0.9s
Checking:
 * debian-11.2.0-i386-net…clone_chunk.012_5ixyt8: checking
Transferring:
 *                debian-11.2.0-i386-netinst.iso:100% /470Mi, 2.911Mi/s, -

2022/01/23 06:25:51 INFO  : debian-11.2.0-i386-netinst.iso.rclone_chunk.012_5ixyt8: Moved (server-side) to: debian-11.2.0-i386-netinst.iso.rclone_chunk.012
2022/01/23 06:25:51 INFO  : debian-11.2.0-i386-netinst.iso.rclone_chunk.013_5ixyt8: Moved (server-side) to: debian-11.2.0-i386-netinst.iso.rclone_chunk.013
2022/01/23 06:25:51 INFO  : debian-11.2.0-i386-netinst.iso.rclone_chunk.014_5ixyt8: Moved (server-side) to: debian-11.2.0-i386-netinst.iso.rclone_chunk.014
2022/01/23 06:25:52 INFO  : debian-11.2.0-i386-netinst.iso.rclone_chunk.015_5ixyt8: Moved (server-side) to: debian-11.2.0-i386-netinst.iso.rclone_chunk.015
2022/01/23 06:25:52 INFO  : debian-11.2.0-i386-netinst.iso: Copied (new)
2022/01/23 06:25:57 INFO  : debian-11.2.0-i386-netinst.iso: Deleted
2022/01/23 06:25:57 INFO  :
Transferred:      470.115 MiB / 470.115 MiB, 100%, 1.897 MiB/s, ETA 0s
Checks:                17 / 17, 100%
Deleted:                1 (files), 0 (dirs)
Renamed:               16
Transferred:            1 / 1, 100%
Elapsed time:       2m7.9s

2022/01/23 06:25:57 DEBUG : 5 go routines active

How to use GitHub

  • Please use the 👍 reaction to show that you are affected by the same issue.
  • Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
  • Subscribe to receive notifications on status change and new comments.
@ivandeex ivandeex self-assigned this Jan 23, 2022
@ivandeex ivandeex added this to the v1.59 milestone Jan 23, 2022
@ivandeex ivandeex modified the milestones: v1.59, v1.60 Feb 13, 2022
@ncw ncw modified the milestones: v1.60, Known Problem Dec 5, 2022
@Jiayou-Chao
Copy link

Hello, I read this paragraph on the chunker documentation page:

Also, chunker will reject a server-side copy or move operation if source and destination hashsum types are different resulting in the extra network bandwidth, too. In some rare cases this may be undesired, so chunker provides two optional choices: sha1quick and md5quick.

So it seems it is caused by the hash type? Maybe you can set it to sha1quick or md5quick and try again.

@eharris
Copy link
Contributor

eharris commented Apr 7, 2024

Seems like this is the kind of thing that #7728 would help quite a bit with debugging/visibility.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ivandeex ivandeex

Labels
bug Remote: Chunker
Projects
None yet
Milestone
Known Problem
Development

No branches or pull requests
6 participants
@ncw @Animosity022 @ivandeex @eharris @jdr85 @Jiayou-Chao