--b2-hard-delete creating/keeping versions #2403

atomtigerzoo · 2018-07-18T12:57:42Z

What is the problem you are having with rclone?

The command

rclone sync test/ remoteName:bucketName --fast-list --b2-hard-delete

is still creating versions on B2, not removing old files.

I changed the index.html several times, then issuing the command above. Always getting a new version of the file. Lifecycle settings of the bucket are set keep only the current version.

What is your rclone version (eg output from `rclone -V`)

rclone v1.42

os/arch: linux/amd64
go version: go1.10.1

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Debian stable

Which cloud storage system are you using? (eg Google Drive)

B2

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

rclone sync test/ remoteName:bucketName --fast-list --b2-hard-delete

A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)

2018/07/18 14:53:44 DEBUG : index.html: Modification times differ by -3m41.864114454s: 2018-07-18 14:53:37.960114454 +0200 CEST, 2018-07-18 12:49:56.096 +0000 UTC
2018/07/18 14:53:44 DEBUG : index.html: SHA-1 = 39422c5a9980b4e843db2abd0a88b71adafd6b38 (Local file system at /home/user/test)
2018/07/18 14:53:44 DEBUG : index.html: SHA-1 = fa48f8eeaae1769c42336c655aca832f7c7e6a85 (B2 bucket bucketName)
2018/07/18 14:53:44 DEBUG : index.html: SHA-1 differ
2018/07/18 14:53:44 INFO  : B2 bucket bucketName: Waiting for checks to finish
2018/07/18 14:53:44 INFO  : B2 bucket bucketName: Waiting for transfers to finish
2018/07/18 14:53:46 INFO  : index.html: Copied (replaced existing)
2018/07/18 14:53:46 INFO  : Waiting for deletions to finish
2018/07/18 14:53:46 INFO  : 
Transferred:     45 Bytes (8 Bytes/s)
Errors:                 0
Checks:                 2
Transferred:            1
Elapsed time:        5.2s

The text was updated successfully, but these errors were encountered:

ncw · 2018-07-18T22:08:13Z

I would say rclone is calling the correct API...

First non --hard-delete

$ echo "hello" | rclone rcat b2:rclone-test1/hello1.txt
$ rclone -vv --dump bodies deletefile b2:rclone-test1/hello1.txt
2018/07/18 23:03:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/18 23:03:54 DEBUG : HTTP REQUEST (req 0xc4200ee200)
2018/07/18 23:03:54 DEBUG : POST /b2api/v1/b2_hide_file HTTP/1.1
Host: api001.backblazeb2.com
User-Agent: rclone/v1.42-053-g77b1eaef-drive-untrash
Content-Length: 63
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

{"bucketId":"7fd2ed1360fdeb1b6f180e1b","fileName":"hello1.txt"}
2018/07/18 23:03:54 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/18 23:03:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/18 23:03:55 DEBUG : HTTP RESPONSE (req 0xc4200ee200)
2018/07/18 23:03:55 DEBUG : HTTP/1.1 200 
Content-Length: 349
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Wed, 18 Jul 2018 22:03:54 GMT

{
  "action": "hide",
  "contentLength": 0,
  "contentSha1": "da39a3ee5e6b4b0d3255bfef95601890afd80709",
  "contentType": "application/x-bz-hide-marker",
  "fileId": "4_z7fd2ed1360fdeb1b6f180e1b_f118f4a90a9b3ed7f_d20180718_m220354_c001_v0001104_t0046",
  "fileInfo": {},
  "fileName": "hello1.txt",
  "size": 0,
  "uploadTimestamp": 1531951434000
}
2018/07/18 23:03:55 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/18 23:03:55 INFO  : hello1.txt: Deleted
2018/07/18 23:03:55 DEBUG : 6 go routines active
2018/07/18 23:03:55 DEBUG : rclone: Version "v1.42-053-g77b1eaef-drive-untrash" finishing with parameters ["rclone" "-vv" "--dump" "bodies" "deletefile" "b2:rclone-test1/hello1.txt"]

Now with --hard-delete

$ echo "hello" | rclone rcat b2:rclone-test1/hello2.txt
$ rclone -vv --dump bodies --b2-hard-delete deletefile b2:rclone-test1/hello2.txt
2018/07/18 23:04:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/18 23:04:23 DEBUG : HTTP REQUEST (req 0xc4204ce100)
2018/07/18 23:04:23 DEBUG : POST /b2api/v1/b2_delete_file_version HTTP/1.1
Host: api001.backblazeb2.com
User-Agent: rclone/v1.42-053-g77b1eaef-drive-untrash
Content-Length: 120
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

{"fileId":"4_z7fd2ed1360fdeb1b6f180e1b_f1163fc0adff0ef4e_d20180718_m220408_c001_v0001091_t0059","fileName":"hello2.txt"}
2018/07/18 23:04:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/18 23:04:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/18 23:04:23 DEBUG : HTTP RESPONSE (req 0xc4204ce100)
2018/07/18 23:04:23 DEBUG : HTTP/1.1 200 
Content-Length: 130
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Wed, 18 Jul 2018 22:04:23 GMT

{
  "fileId": "4_z7fd2ed1360fdeb1b6f180e1b_f1163fc0adff0ef4e_d20180718_m220408_c001_v0001091_t0059",
  "fileName": "hello2.txt"
}
2018/07/18 23:04:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/18 23:04:23 INFO  : hello2.txt: Deleted
2018/07/18 23:04:23 DEBUG : 6 go routines active
2018/07/18 23:04:23 DEBUG : rclone: Version "v1.42-053-g77b1eaef-drive-untrash" finishing with parameters ["rclone" "-vv" "--dump" "bodies" "--b2-hard-delete" "deletefile" "b2:rclone-test1/hello2.txt"]
$

Can you make a little sequence like that which shows the problem - maybe using rclone ls --b2-versions to show the versions you aren't expecting?

atomtigerzoo · 2018-07-19T14:16:25Z

Why are you using deletefile? I want to upload/sync my folder not delete a file :) Or am I understanding it wrong? :)

Here is another extensive test with logs:

~$ echo "Hi!" > test/hello.txt

~$ rclone -vv --dump bodies --fast-list --b2-hard-delete sync test/ b2:myBukit
2018/07/19 16:04:09 DEBUG : rclone: Version "v1.42" starting with parameters ["rclone" "-vv" "--dump" "bodies" "--fast-list" "--b2-hard-delete" "sync" "test/" "b2:myBukit"]
2018/07/19 16:04:11 DEBUG : Using config file from "/home/username/.config/rclone/rclone.conf"
2018/07/19 16:04:11 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:04:11 DEBUG : HTTP REQUEST (req 0xc42037ee00)
2018/07/19 16:04:11 DEBUG : GET /b2api/v1/b2_authorize_account HTTP/1.1
Host: api.backblazeb2.com
User-Agent: rclone/v1.42
Authorization: XXXX
Accept-Encoding: gzip

2018/07/19 16:04:11 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:04:12 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:04:12 DEBUG : HTTP RESPONSE (req 0xc42037ee00)
2018/07/19 16:04:12 DEBUG : HTTP/1.1 200
Content-Length: 446
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Thu, 19 Jul 2018 14:04:12 GMT

{
  "absoluteMinimumPartSize": 5000000,
  "accountId": "XXXX",
  "allowed": {
	"bucketId": null,
	"capabilities": [
	  "all"
	],
	"namePrefix": null
  },
  "apiUrl": "https://api001.backblazeb2.com",
  "authorizationToken": "XXXX",
  "downloadUrl": "https://f001.backblazeb2.com",
  "minimumPartSize": 100000000,
  "recommendedPartSize": 100000000
}
2018/07/19 16:04:12 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:04:12 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:04:12 DEBUG : HTTP REQUEST (req 0xc4200d2400)
2018/07/19 16:04:12 DEBUG : POST /b2api/v1/b2_list_buckets HTTP/1.1
Host: api001.backblazeb2.com
User-Agent: rclone/v1.42
Content-Length: 28
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

{"accountId":"XXXX"}
2018/07/19 16:04:12 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:04:13 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:04:13 DEBUG : HTTP RESPONSE (req 0xc4200d2400)
2018/07/19 16:04:13 DEBUG : HTTP/1.1 200
Content-Length: 20
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Thu, 19 Jul 2018 14:04:13 GMT

{
  "buckets": []
}
2018/07/19 16:04:13 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:04:13 INFO  : B2 bucket myBukit: Waiting for checks to finish
2018/07/19 16:04:13 INFO  : B2 bucket myBukit: Waiting for transfers to finish
2018/07/19 16:04:13 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:04:13 DEBUG : HTTP REQUEST (req 0xc4200d2800)
2018/07/19 16:04:13 DEBUG : POST /b2api/v1/b2_create_bucket HTTP/1.1
Host: api001.backblazeb2.com
User-Agent: rclone/v1.42
Content-Length: 83
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

{"accountId":"XXXX","bucketName":"myBukit","bucketType":"allPrivate"}
2018/07/19 16:04:13 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:04:13 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:04:13 DEBUG : HTTP RESPONSE (req 0xc4200d2800)
2018/07/19 16:04:13 DEBUG : HTTP/1.1 200
Content-Length: 219
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Thu, 19 Jul 2018 14:04:14 GMT

{
  "accountId": "XXXX",
  "bucketId": "XXXX",
  "bucketInfo": {},
  "bucketName": "myBukit",
  "bucketType": "allPrivate",
  "corsRules": [],
  "lifecycleRules": [],
  "revision": 2
}
2018/07/19 16:04:13 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:04:13 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:04:13 DEBUG : HTTP REQUEST (req 0xc4201acb00)
2018/07/19 16:04:13 DEBUG : POST /b2api/v1/b2_get_upload_url HTTP/1.1
Host: api001.backblazeb2.com
User-Agent: rclone/v1.42
Content-Length: 39
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

{"bucketId":"XXXX"}
2018/07/19 16:04:13 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:04:13 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:04:13 DEBUG : HTTP RESPONSE (req 0xc4201acb00)
2018/07/19 16:04:13 DEBUG : HTTP/1.1 200
Content-Length: 275
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Thu, 19 Jul 2018 14:04:14 GMT

{
  "authorizationToken": "XXXX",
  "bucketId": "XXXX",
  "uploadUrl": "https://pod-000-1099-01.backblaze.com/b2api/v1/b2_upload_file/XXXX/XXXX"
}
2018/07/19 16:04:13 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:04:13 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:04:13 DEBUG : HTTP REQUEST (req 0xc42037e700)
2018/07/19 16:04:13 DEBUG : POST /b2api/v1/b2_upload_file/XXXX/XXXX HTTP/1.1
Host: pod-000-1099-01.backblaze.com
User-Agent: rclone/v1.42
Content-Length: 4
Authorization: XXXX
Content-Type: text/plain; charset=utf-8
X-Bz-Content-Sha1: eb8e0e5fcbd4675e9e6fc6f770a170e56bd5923b
X-Bz-File-Name: hello.txt
X-Bz-Info-Src_last_modified_millis: 1532008517218
Accept-Encoding: gzip

Hi!
2018/07/19 16:04:13 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:04:14 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:04:14 DEBUG : HTTP RESPONSE (req 0xc42037e700)
2018/07/19 16:04:14 DEBUG : HTTP/1.1 200
Content-Length: 458
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Thu, 19 Jul 2018 14:04:15 GMT

{
  "accountId": "XXXX",
  "action": "upload",
  "bucketId": "XXXX",
  "contentLength": 4,
  "contentSha1": "eb8e0e5fcbd4675e9e6fc6f770a170e56bd5923b",
  "contentType": "text/plain; charset=utf-8",
  "fileId": "4_zXXXX_f1019db06cec50136_d20180719_m140415_XXXX",
  "fileInfo": {
	"src_last_modified_millis": "1532008517218"
  },
  "fileName": "hello.txt",
  "uploadTimestamp": 1532009055000
}
2018/07/19 16:04:14 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:04:14 INFO  : hello.txt: Copied (new)
2018/07/19 16:04:14 INFO  : Waiting for deletions to finish
2018/07/19 16:04:14 INFO  :
Transferred:      4 Bytes (0 Bytes/s)
Errors:                 0
Checks:                 0
Transferred:            1
Elapsed time:        5.8s

2018/07/19 16:04:14 DEBUG : 10 go routines active
2018/07/19 16:04:14 DEBUG : rclone: Version "v1.42" finishing with parameters ["rclone" "-vv" "--dump" "bodies" "--fast-list" "--b2-hard-delete" "sync" "test/" "b2:myBukit"]

~$ rclone --fast-list --b2-versions ls b2:myBukit
4 hello.txt

So we have one file in there. Let's edit it:

~$ echo "Is it me you're looking for?" > test/hello.txt

	~$ rclone -vv --dump bodies --fast-list --b2-hard-delete sync test/ b2:myBuckit
2018/07/19 16:10:20 DEBUG : rclone: Version "v1.42" starting with parameters ["rclone" "-vv" "--dump" "bodies" "--fast-list" "--b2-hard-delete" "sync" "test/" "b2:myBuckit"]
Enter configuration password:
password:
2018/07/19 16:10:23 DEBUG : Using config file from "/home/user/.config/rclone/rclone.conf"
2018/07/19 16:10:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:10:23 DEBUG : HTTP REQUEST (req 0xc42033b100)
2018/07/19 16:10:23 DEBUG : GET /b2api/v1/b2_authorize_account HTTP/1.1
Host: api.backblazeb2.com
User-Agent: rclone/v1.42
Authorization: XXXX
Accept-Encoding: gzip

2018/07/19 16:10:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:10:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:10:23 DEBUG : HTTP RESPONSE (req 0xc42033b100)
2018/07/19 16:10:23 DEBUG : HTTP/1.1 200
Content-Length: 446
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Thu, 19 Jul 2018 14:10:24 GMT

{
  "absoluteMinimumPartSize": 5000000,
  "accountId": "XXXX",
  "allowed": {
	"bucketId": null,
	"capabilities": [
	  "all"
	],
	"namePrefix": null
  },
  "apiUrl": "https://api001.backblazeb2.com",
  "authorizationToken": "XXXX",
  "downloadUrl": "https://f001.backblazeb2.com",
  "minimumPartSize": 100000000,
  "recommendedPartSize": 100000000
}
2018/07/19 16:10:23 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:10:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:10:23 DEBUG : HTTP REQUEST (req 0xc4200d2700)
2018/07/19 16:10:23 DEBUG : POST /b2api/v1/b2_list_buckets HTTP/1.1
Host: api001.backblazeb2.com
User-Agent: rclone/v1.42
Content-Length: 28
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

{"accountId":"XXXX"}
2018/07/19 16:10:23 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:10:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:10:24 DEBUG : HTTP RESPONSE (req 0xc4200d2700)
2018/07/19 16:10:24 DEBUG : HTTP/1.1 200
Content-Length: 282
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Thu, 19 Jul 2018 14:10:24 GMT

{
  "buckets": [
	{
	  "accountId": "XXXX",
	  "bucketId": "XXXX",
	  "bucketInfo": {},
	  "bucketName": "myBuckit",
	  "bucketType": "allPrivate",
	  "corsRules": [],
	  "lifecycleRules": [],
	  "revision": 2
	}
  ]
}
2018/07/19 16:10:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:10:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:10:24 DEBUG : HTTP REQUEST (req 0xc420224100)
2018/07/19 16:10:24 DEBUG : POST /b2api/v1/b2_list_file_names HTTP/1.1
Host: api001.backblazeb2.com
User-Agent: rclone/v1.42
Content-Length: 59
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

{"bucketId":"XXXX","maxFileCount":1000}
2018/07/19 16:10:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:10:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:10:24 DEBUG : HTTP RESPONSE (req 0xc420224100)
2018/07/19 16:10:24 DEBUG : HTTP/1.1 200
Content-Length: 495
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Thu, 19 Jul 2018 14:10:24 GMT

{
  "files": [
	{
	  "action": "upload",
	  "contentLength": 4,
	  "contentSha1": "eb8e0e5fcbd4675e9e6fc6f770a170e56bd5923b",
	  "contentType": "text/plain; charset=utf-8",
	  "fileId": "4_zXXXX_f1019db06cec50136_d20180719_m140415_c001_v0001099_t0019",
	  "fileInfo": {
		"src_last_modified_millis": "1532008517218"
	  },
	  "fileName": "hello.txt",
	  "size": 4,
	  "uploadTimestamp": 1532009055000
	}
  ],
  "nextFileName": null
}
2018/07/19 16:10:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:10:24 DEBUG : hello.txt: Sizes differ (src 30 vs dst 4)
2018/07/19 16:10:24 INFO  : B2 bucket myBuckit: Waiting for checks to finish
2018/07/19 16:10:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:10:24 DEBUG : HTTP REQUEST (req 0xc420224300)
2018/07/19 16:10:24 DEBUG : POST /b2api/v1/b2_get_upload_url HTTP/1.1
Host: api001.backblazeb2.com
User-Agent: rclone/v1.42
Content-Length: 39
Authorization: XXXX
Content-Type: application/json
Accept-Encoding: gzip

{"bucketId":"XXXX"}
2018/07/19 16:10:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:10:24 INFO  : B2 bucket myBuckit: Waiting for transfers to finish
2018/07/19 16:10:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:10:24 DEBUG : HTTP RESPONSE (req 0xc420224300)
2018/07/19 16:10:24 DEBUG : HTTP/1.1 200
Content-Length: 275
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Thu, 19 Jul 2018 14:10:25 GMT

{
  "authorizationToken": "XXXX",
  "bucketId": "XXXX",
  "uploadUrl": "https://pod-000-1105-06.backblaze.com/b2api/v1/b2_upload_file/XXXX/XXXX"
}
2018/07/19 16:10:24 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:10:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:10:24 DEBUG : HTTP REQUEST (req 0xc420224700)
2018/07/19 16:10:24 DEBUG : POST /b2api/v1/b2_upload_file/XXXX/XXXX HTTP/1.1
Host: pod-000-1105-06.backblaze.com
User-Agent: rclone/v1.42
Content-Length: 30
Authorization: XXXX
Content-Type: text/plain; charset=utf-8
X-Bz-Content-Sha1: a4d15a4ee86b368b7b99c3ed5481d4f8557e90af
X-Bz-File-Name: hello.txt
X-Bz-Info-Src_last_modified_millis: 1532009379866
Accept-Encoding: gzip

 Is it me you're looking for?
2018/07/19 16:10:24 DEBUG : >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2018/07/19 16:10:26 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:10:26 DEBUG : HTTP RESPONSE (req 0xc420224700)
2018/07/19 16:10:26 DEBUG : HTTP/1.1 200
Content-Length: 459
Cache-Control: max-age=0, no-cache, no-store
Content-Type: application/json;charset=UTF-8
Date: Thu, 19 Jul 2018 14:10:26 GMT

{
  "accountId": "XXXX",
  "action": "upload",
  "bucketId": "XXXX",
  "contentLength": 30,
  "contentSha1": "a4d15a4ee86b368b7b99c3ed5481d4f8557e90af",
  "contentType": "text/plain; charset=utf-8",
  "fileId": "4_zXXXX_f106bc865bb835763_d20180719_m141026_XXXX",
  "fileInfo": {
	"src_last_modified_millis": "1532009379866"
  },
  "fileName": "hello.txt",
  "uploadTimestamp": 1532009426000
}
2018/07/19 16:10:26 DEBUG : <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
2018/07/19 16:10:26 INFO  : hello.txt: Copied (replaced existing)
2018/07/19 16:10:26 INFO  : Waiting for deletions to finish
2018/07/19 16:10:26 INFO  :
Transferred:     30 Bytes (4 Bytes/s)
Errors:                 0
Checks:                 1
Transferred:            1
Elapsed time:        6.1s

2018/07/19 16:10:26 DEBUG : 9 go routines active
2018/07/19 16:10:26 DEBUG : rclone: Version "v1.42" finishing with parameters ["rclone" "-vv" "--dump" "bodies" "--fast-list" "--b2-hard-delete" "sync" "test/" "b2:myBuckit"]

~$ rclone --fast-list --b2-versions ls b2:myBuckit
 4 hello-v2018-07-19-140415-000.txt
 30 hello.txt

And here we have the old version of hello.txt :/

Or am I just using the commands all wrong?

atomtigerzoo · 2018-07-19T14:17:41Z

Sorry, wrong button 😕

ncw · 2018-07-20T15:02:12Z

Ah, I see what you mean - great demo! I think if you'd deleted hello.txt instead of giving it new content then rclone would have done what you expected.

So what is happening is that when rclone uploads a new hello.txt, b2 is creating a new version.

At no time does rclone issue a delete so --b2-hard-delete doesn't come into play.

So I think the --b2-hard-delete flag is doing what it says, but not what you (and likely other users) expect.

What do you think it should do? When rclone comes to update a file that already exists, do a hard delete on the existing file to stop b2 creating a new version? Maybe that should be a different flag --b2-no-versions?

I think the upload api always creates a new version so if we don't want to do that then we'll have to delete any existing file.

atomtigerzoo · 2018-07-21T10:11:48Z

Ah ok, I got the docs at versioning wrong then. Sorry for that! I thought it means that also when uploading new or changed files, there won't be hidden ones and only one version.

I also found the info that if you set the lifecycle rules to "Keep Only Last Version" in the B2 docs means that it will stick around for one day: "This keeps only the most current version of a file. The previous version of the file is “hidden” for one day and then deleted."

Since I am planning to keep versions on my own, an option would be great to delete older versions or hidden files completely. Maybe with the option --b2-no-versions you are suggesting.

Thanks for the clarification and I hope to see that flag some day :)

dlong500 · 2018-08-10T05:18:16Z

Would deleting changed files during a sync use additional B2 transactions? If so then it might be more cost effective to just let B2 expire the hidden files in a day rather than forcibly delete them.

ncw · 2018-08-10T10:37:53Z

@dlong500 wrote:

Would deleting changed files during a sync use additional B2 transactions? If so then it might be more cost effective to just let B2 expire the hidden files in a day rather than forcibly delete them.

deletes are free at b2. So rclone would have to know the ID of the existing file (which it should already) to delete it afterwards.

I think if it were me I'd stick with the delete after 1 day as that gives you 24 hours to recover stuff if stuff goes wrong.

ivandeex · 2021-02-09T16:18:56Z

@atomtigerzoo @dlong500 This problem is reported against obsolete rclone version. Can you reproduce it with rclone 1.54?

ivandeex · 2021-03-15T22:50:55Z

Closing answered questions

LinuxOnTheDesktop · 2021-12-16T15:58:24Z

@ivandeex

Can you reproduce it with rclone 1.54?

I can, on rclone v1.57.0 - please see here on the Rclone forum.

EDITED.

LinuxOnTheDesktop · 2021-12-18T02:48:55Z

Cf. #462 and #1547. The latter suggests that progress with rclone on this front got as far, only, as a beta.

atomtigerzoo closed this as completed Jul 19, 2018

atomtigerzoo reopened this Jul 19, 2018

ncw added bug enhancement Remote: Backblaze B2 labels Jul 20, 2018

ncw added this to the v1.43 milestone Jul 20, 2018

ncw modified the milestones: v1.43, Help Wanted Sep 2, 2018

ivandeex added waiting for reply... versioning labels Feb 9, 2021

ivandeex closed this as completed Mar 15, 2021

LinuxOnTheDesktop mentioned this issue Dec 18, 2021

b2: clean previous file versions #462

Closed

LinuxOnTheDesktop mentioned this issue Dec 22, 2021

Ability to cleanup individual old file versions in B2 #1626

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

--b2-hard-delete creating/keeping versions #2403

--b2-hard-delete creating/keeping versions #2403

atomtigerzoo commented Jul 18, 2018

ncw commented Jul 18, 2018

atomtigerzoo commented Jul 19, 2018 •

edited

atomtigerzoo commented Jul 19, 2018

ncw commented Jul 20, 2018

atomtigerzoo commented Jul 21, 2018

dlong500 commented Aug 10, 2018

ncw commented Aug 10, 2018

ivandeex commented Feb 9, 2021

ivandeex commented Mar 15, 2021

LinuxOnTheDesktop commented Dec 16, 2021 •

edited

LinuxOnTheDesktop commented Dec 18, 2021

--b2-hard-delete creating/keeping versions #2403

--b2-hard-delete creating/keeping versions #2403

Comments

atomtigerzoo commented Jul 18, 2018

What is the problem you are having with rclone?

What is your rclone version (eg output from rclone -V)

Which OS you are using and how many bits (eg Windows 7, 64 bit)

Which cloud storage system are you using? (eg Google Drive)

The command you were trying to run (eg rclone copy /tmp remote:tmp)

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

ncw commented Jul 18, 2018

atomtigerzoo commented Jul 19, 2018 • edited

atomtigerzoo commented Jul 19, 2018

ncw commented Jul 20, 2018

atomtigerzoo commented Jul 21, 2018

dlong500 commented Aug 10, 2018

ncw commented Aug 10, 2018

ivandeex commented Feb 9, 2021

ivandeex commented Mar 15, 2021

LinuxOnTheDesktop commented Dec 16, 2021 • edited

LinuxOnTheDesktop commented Dec 18, 2021

What is your rclone version (eg output from `rclone -V`)

The command you were trying to run (eg `rclone copy /tmp remote:tmp`)

A log from the command with the `-vv` flag (eg output from `rclone -vv copy /tmp remote:tmp`)

atomtigerzoo commented Jul 19, 2018 •

edited

LinuxOnTheDesktop commented Dec 16, 2021 •

edited