Hasher: copyto and moveto do not always save hash

[Jwink3101]

The associated forum post URL from https://forum.rclone.org

https://forum.rclone.org/t/hasher-bugs-1-copyto-moveto-do-not-always-works-and-2-hashsum-download-does-not-always-save-hash/30788

What is the problem you are having with rclone?

Using copyto and moveto does not always capture the hash. Especially on a second run. It may have to do with whether a file is present already

What is your rclone version (output from rclone version)

rclone v1.58.1
- os/version: darwin 10.15.7 (64 bit)
- os/kernel: 19.6.0 (x86_64)
- os/type: darwin
- os/arch: amd64
- go/version: go1.17.9
- go/linking: dynamic
- go/tags: cmount

Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

macoS 64 bit x86

Which cloud storage system are you using? (e.g. Google Drive)

hasher on crypt on local (testing with local for now)

The command you were trying to run (e.g. rclone copy /tmp remote:tmp)

See narrative below

A log from the command with the -vv flag (e.g. output from rclone -vv copy /tmp remote:tmp)

see narrative below.

---

Narrative

Clean everything up. Note that this password is just playing so it is not sensitive. We use crypt because (a) I will eventually be using it and (b) it mimics a remote without hashes

export RCLONE_CACHE_DIR='cache'
export RCLONE_TEMP_DIR='tmp'

rm -r $RCLONE_CACHE_DIR 2> /dev/null
mkdir -p $RCLONE_CACHE_DIR
mkdir -p $RCLONE_TEMP_DIR

rm -rf remote/ 2>/dev/null
mkdir -p remote/

export RCLONE_CONFIG=cfg1

cat << EOF > cfg1
[base]
type = alias
remote = remote/

[crypt]
type = crypt
remote = base:
filename_encryption = off
directory_name_encryption = false
password = ZWeBDBLmxXZNusb2tQ73V0Ha9AM
password2 = b8qxr_wqSTyvDS7-_EdR3_PM

[hasher]
type = hasher
remote = crypt:
EOF

Demonstrate the issue

echo "testmove" > move.txt && rclone moveto move.txt hasher:moved.txt
echo "testcopy" > copy.txt && rclone copyto copy.txt hasher:copied.txt
rclone hashsum SHA1 hasher:

echo "testmove" > move.txt && rclone moveto move.txt hasher:moved.txt
echo "testcopy" > copy.txt && rclone copyto copy.txt hasher:copied.txt
rclone hashsum SHA1 hasher: -vv

returns

9c33f710a061db7210f2d77cd98780f5d1e797a3  moved.txt
d44b893ca7b34450022a104ea4dc5304dfb01e2a  copied.txt
2022/05/14 16:51:06 DEBUG : Setting --config "cfg1" from environment variable RCLONE_CONFIG="cfg1"
2022/05/14 16:51:06 DEBUG : Setting --cache-dir "cache" from environment variable RCLONE_CACHE_DIR="cache"
2022/05/14 16:51:06 DEBUG : Setting --temp-dir "tmp" from environment variable RCLONE_TEMP_DIR="tmp"
2022/05/14 16:51:06 DEBUG : rclone: Version "v1.58.1" starting with parameters ["rclone" "hashsum" "SHA1" "hasher:" "-vv"]
2022/05/14 16:51:06 DEBUG : Creating backend with remote "hasher:"
2022/05/14 16:51:06 DEBUG : Using config file from "/<REDACTEDFULLPATH>/rclone_hasher/cfg1"
2022/05/14 16:51:06 INFO  : Hasher is EXPERIMENTAL!
2022/05/14 16:51:06 DEBUG : Creating backend with remote "crypt:"
2022/05/14 16:51:06 DEBUG : Creating backend with remote "base:"
2022/05/14 16:51:06 DEBUG : Creating backend with remote "remote/"
2022/05/14 16:51:06 DEBUG : fs cache: renaming cache item "remote/" to be canonical "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:51:06 DEBUG : fs cache: renaming cache item "base:" to be canonical "/<REDACTEDFULLPATH>/rclone_hasher/remote"
2022/05/14 16:51:06 DEBUG : hasher::hasher:: Groups by usage: cached [md5, sha1], passed [], auto [md5, sha1], slow [], supported [md5, sha1]
2022/05/14 16:51:06 DEBUG : crypt~hasher.bolt: Opened for reading in 73.694µs
2022/05/14 16:51:06 DEBUG : moved.txt: getHash: no record
                                          moved.txt
2022/05/14 16:51:06 DEBUG : copied.txt: getHash: no record
                                          copied.txt
2022/05/14 16:51:06 DEBUG : 5 go routines active

Nothing!

How to use GitHub

• Please use the 👍 reaction to show that you are affected by the same issue.
• Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
• Subscribe to receive notifications on status change and new comments.

[ncw]

@ivandeex what do you make of this?