cache backend: option to cache only metadata (deprecated) #4097

harryqt · 2020-03-29T18:13:01Z

Update by ivandeex:
This description is outdated.
See #4097 (comment) and https://forum.rclone.org/t/how-to-cache-only-metadata-on-cache-remote/14846/5

[src]
type = drive
scope = drive
service_account_file = {redacted}
team_drive = {redacted}

[cachesrc]
type = cache
remote = src:
chunk_size = 0
chunk_clean_interval = 24h
info_age = 24h

[dst]
type = drive
scope = drive
service_account_file = {redacted}
team_drive = {redacted}

[cachedst]
type = cache
remote = dst:
chunk_size = 0
chunk_clean_interval = 24h
info_age = 24h

rclone sync cachesrc: cachedst: --fast-list --tpslimit=6 --drive-stop-on-upload-limit --drive-server-side-across-configs -vv

Instead of doing server side copy, Rclone doing normal copy.

Update:

Running the following command throwing errors.

rclone sync cachesrc: cachedst: --drive-stop-on-upload-limit --drive-server-side-across-configs -vv

C:\Users\HARRY>rclone sync cachesrc: cachedst: --drive-stop-on-upload-limit --drive-server-side-across-configs -vv
2020/03/30 11:02:08 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rclone" "sync" "cachesrc:" "cachedst:" "--drive-stop-on-upload-limit" "--drive-server-side-across-configs" "-vv"]
2020/03/30 11:02:08 DEBUG : Using config file from "C:\\Users\\HARRY\\.config\\rclone\\rclone.conf"
2020/03/30 11:02:08 DEBUG : cachesrc: wrapped src: at root
2020/03/30 11:02:08 INFO  : cachesrc: Cache DB path: C:\Users\HARRY\AppData\Local\rclone\cache-backend\cachesrc.db
2020/03/30 11:02:08 INFO  : cachesrc: Cache chunk path: C:\Users\HARRY\AppData\Local\rclone\cache-backend\cachesrc
2020/03/30 11:02:08 INFO  : cachesrc: Chunk Memory: true
2020/03/30 11:02:08 INFO  : cachesrc: Chunk Size: 0
2020/03/30 11:02:08 INFO  : cachesrc: Chunk Total Size: 10G
2020/03/30 11:02:08 INFO  : cachesrc: Chunk Clean Interval: 1d
2020/03/30 11:02:08 INFO  : cachesrc: Workers: 4
2020/03/30 11:02:08 INFO  : cachesrc: File Age: 1d
2020/03/30 11:02:08 DEBUG : Adding path "cache/expire" to remote control registry
2020/03/30 11:02:08 DEBUG : Adding path "cache/stats" to remote control registry
2020/03/30 11:02:08 DEBUG : Adding path "cache/fetch" to remote control registry
2020/03/30 11:02:08 DEBUG : cachedst: wrapped dst: at root
2020/03/30 11:02:08 INFO  : cachedst: Cache DB path: C:\Users\HARRY\AppData\Local\rclone\cache-backend\cachedst.db
2020/03/30 11:02:08 INFO  : cachedst: Cache chunk path: C:\Users\HARRY\AppData\Local\rclone\cache-backend\cachedst
2020/03/30 11:02:08 INFO  : cachedst: Chunk Memory: true
2020/03/30 11:02:08 INFO  : cachedst: Chunk Size: 0
2020/03/30 11:02:08 INFO  : cachedst: Chunk Total Size: 10G
2020/03/30 11:02:08 INFO  : cachedst: Chunk Clean Interval: 1d
2020/03/30 11:02:08 INFO  : cachedst: Workers: 4
2020/03/30 11:02:08 INFO  : cachedst: File Age: 1d
2020/03/30 11:02:08 DEBUG : Adding path "cache/expire" to remote control registry
2020/03/30 11:02:08 DEBUG : Adding path "cache/stats" to remote control registry
2020/03/30 11:02:08 DEBUG : Adding path "cache/fetch" to remote control registry
2020/03/30 11:02:08 DEBUG : Cache remote cachedst:: list ''
2020/03/30 11:02:08 DEBUG : Cache remote cachesrc:: list ''
2020/03/30 11:02:08 DEBUG : : list: warm 1 from cache for: , expiring on: 2020-03-31 11:00:48.8395238 +0530 IST
2020/03/30 11:02:08 DEBUG : : list: cached entries: [generatepress_child.zip]
2020/03/30 11:02:08 DEBUG : : list: warm 8 from cache for: , expiring on: 2020-03-31 10:58:36.1859809 +0530 IST
2020/03/30 11:02:08 DEBUG : : list: cached entries: [21.Bridges.2019.1080p.10bit.BluRay.6CH.x265.HEVC-PSA.mkv Bombshell.20
19.1080p.10bit.BluRay.6CH.x265.HEVC-PSA.mkv WP_Rocket_v3.5.1_Final.zip generatepress_child.zip gp-premium-1.9.1.zip rclone
-beta-latest-linux-amd64.zip themeforest-BW1E6RdJ-masterstudy-education-center-wordpress-theme.zip themeforest-VaFPTZIE-ma
sterstudy-education-center-wordpress-theme.zip]
2020/03/30 11:02:08 DEBUG : generatepress_child.zip: Size and modification time the same (differ by 0s, within tolerance 1
ms)
2020/03/30 11:02:08 DEBUG : generatepress_child.zip: Unchanged skipping
2020/03/30 11:02:08 INFO  : Cache remote cachedst:: Waiting for checks to finish
2020/03/30 11:02:08 INFO  : Cache remote cachedst:: Waiting for transfers to finish
2020/03/30 11:02:10 DEBUG : WP_Rocket_v3.5.1_Final.zip: moving offset set from 0 to 0
panic: runtime error: integer divide by zero

goroutine 74 [running]:
github.com/rclone/rclone/backend/cache.(*Handle).Seek(0xc000122600, 0x0, 0x0, 0x0, 0x0, 0x0)
        d:/a/rclone/src/github.com/rclone/rclone/backend/cache/handle.go:328 +0x631
github.com/rclone/rclone/backend/cache.(*Object).Open(0xc000144500, 0x1a74e40, 0xc0000bc2c0, 0xc0001b91e0, 0x1, 0x2, 0xc00
0386000, 0x180d58, 0x0, 0xa471bc)
        d:/a/rclone/src/github.com/rclone/rclone/backend/cache/object.go:235 +0x177
github.com/rclone/rclone/fs/operations.(*reOpen).open(0xc000122200, 0xc000122200, 0xc000144500)
        d:/a/rclone/src/github.com/rclone/rclone/fs/operations/reopen.go:83 +0x287
github.com/rclone/rclone/fs/operations.newReOpen(0x1a74e40, 0xc0000bc2c0, 0x1a8a1a0, 0xc000144500, 0xc0003002b0, 0x0, 0xa,
 0x0, 0x0, 0x0, ...)
        d:/a/rclone/src/github.com/rclone/rclone/fs/operations/reopen.go:49 +0x113
github.com/rclone/rclone/fs/operations.Copy(0x1a74e40, 0xc0000bc2c0, 0x1a8a120, 0xc00000aa80, 0x0, 0x0, 0xc0003bc0e0, 0x1a
, 0x1a8a1a0, 0xc000144500, ...)
        d:/a/rclone/src/github.com/rclone/rclone/fs/operations/operations.go:399 +0xf3c
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove(0xc00000a000, 0x1a74e40, 0xc0000bc2c0, 0xc0000c60f0, 0x1a8
a120, 0xc00000aa80, 0xc00000a0d8)
        d:/a/rclone/src/github.com/rclone/rclone/fs/sync/sync.go:331 +0x223
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startTransfers
        d:/a/rclone/src/github.com/rclone/rclone/fs/sync/sync.go:356 +0xb8

The text was updated successfully, but these errors were encountered:

Friday13th87 · 2020-03-30T00:40:26Z

why are u using fast-list / tpslimit / drive-stop-on-upload when u are trying to do a serverside copy?
for me it is working actually and i am working without those options as they are designed for no-server-side copys for my understanding

ncw · 2020-03-31T11:39:45Z

It is the setting of chunk_size = 0 which is causing this problem.

I'm not sure that is a valid value?

harryqt · 2020-03-31T11:42:28Z

I'm not sure that is a valid value?

It's done on purpose. I wants to cache metadata only, already discussed in forum.

ncw · 2020-03-31T11:44:31Z

Ah OK. I'm not sure that is a valid assumption though... Fancy having a rootle through the cache backend code and confirming or denying that? https://github.com/rclone/rclone/tree/master/backend/cache

harryqt · 2020-03-31T12:16:22Z

2020/03/30 11:02:10 DEBUG : WP_Rocket_v3.5.1_Final.zip: moving offset set from 0 to 0
panic: runtime error: integer divide by zero

Hmm. Rclone crashes because it was trying to divide 0 with 0. on handle.go line 208. Actually you already mentioned this.

ncw · 2020-04-01T11:22:24Z

Do you think setting chunk_total_size might be a better way of achieving this?

It seems that cache needs fixing up if chunk_size = 0

harryqt · 2020-04-01T11:45:34Z

My main goal is to do server side copy with the help of cache, so I don't have to sync both src/dest over and over (as suggested by you).

If I use cache-chunk-total-size then isn't rclone gonna download chunks and upload chunks? I think then server side copy won't work. I am testing it though, will update if it works.

Update: Nope. Does not work. I added chunk_total_size = 10m and chunk_size = 1m but still rclone doing normal copy instead of server side copy. The --drive-server-side-across-configs does not comes into effect.

scottmlew · 2020-04-06T23:02:37Z

Although the actual issue appears to be different, your subject describes my open issue about files being reuploaded vs copied/moved locally, so I thought I'd link it to aid anyone perusing this for help

https://forum.rclone.org/t/still-unable-to-move-files-on-gdrive/13921

harryqt · 2020-05-25T02:20:01Z

Any update on this? Has anyone figured out any workaround?

ncw · 2020-05-25T14:36:33Z

We merged a fix which may be of interest 19ff7c9 - I'm not 100% sure this will fix your issue but it might be worth giving the latest beta a go.

harryqt · 2020-05-29T06:24:12Z

Nah, still the same. 😞

ivandeex · 2020-11-15T13:14:43Z

@ncw
Since the cache overlay has been replaced by VFS,
this issue needs triaging.

ivandeex · 2021-03-12T14:52:48Z

@Dibbyo456
Do you experience this issue with 1.54 and VFS?
If yes, please paste below your current config, full command, debug log.
Or submit a new request (this one is obsolete without new debug data).

harryqt · 2021-03-12T16:47:23Z

Do you experience this issue with 1.54 and VFS?

No, I did not check with 1.54 but checked with 1.53.
Although I'm pretty sure it's still broken because chunk_size = 0 is not valid.

ivandeex · 2021-03-12T17:30:01Z

config?
command?
log?

ivandeex · 2021-03-12T17:31:31Z

What is your VFS command?
Or do you still use cache backend?

ivandeex · 2021-03-13T11:03:23Z

@Dibbyo456
If you remove the chunk_size = 0, does the rest work?

ivandeex · 2021-03-13T11:06:53Z

@ncw

I'm interested in resolving this ticket (and closing it out) as this is the last cache related request.
I don't think we will be accepting more cache backend requests,
but this one was opened a year ago before cache was announced deprecated.

harryqt · 2021-03-13T11:24:33Z

I'm interested in resolving this ticket (and closing it out) as this is the last cache related request.

Don't bother. Cache backend is unofficaly deprecated and nick does not recommend anyone to use it.
Instead may be we should do this with VFS becuase AFAIR VFS can't do this either.

harryqt · 2021-03-13T11:27:04Z

Can we cache metadata only in vfs?

ivandeex · 2021-03-13T12:10:56Z

Can we cache metadata only in vfs?

I will investigate that.

ivandeex · 2021-03-13T13:31:05Z

References:

ivandeex · 2021-03-13T13:43:54Z

@Dibbyo456
I opened a new feature request #5123
Please subscribe to it

ivandeex · 2021-03-13T13:45:33Z

This thread is aggravated by deprecated and unrelated discussions.
Closing.

ncw added bug Remote: Cache labels Mar 31, 2020

ncw added this to the v1.52 milestone Mar 31, 2020

ncw modified the milestones: v1.52, v1.53 May 29, 2020

ncw modified the milestones: v1.53, v1.54 Sep 5, 2020

ivandeex added the needs retest label Nov 15, 2020

ncw modified the milestones: v1.54, v1.55 Feb 3, 2021

ivandeex added the waiting for reply... label Mar 12, 2021

ivandeex self-assigned this Mar 13, 2021

ivandeex added VFS / mount question and removed Remote: Cache bug needs retest waiting for reply... labels Mar 13, 2021

ivandeex changed the title ~~Google drive server side copy does not work, when wrapped with cache.~~ vfs: option to cache only metadata Mar 13, 2021

ivandeex changed the title ~~vfs: option to cache only metadata~~ vfs: option to cache only metadata (obsolete) Mar 13, 2021

ivandeex mentioned this issue Mar 13, 2021

vfs: option to cache metadata only #5123

Open

ivandeex closed this as completed Mar 13, 2021

ivandeex changed the title ~~vfs: option to cache only metadata (obsolete)~~ cache backend: option to cache only metadata (deprecated) Mar 13, 2021

ivandeex added bug Remote: Cache and removed VFS / mount question labels Mar 13, 2021

ivandeex removed their assignment Mar 13, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cache backend: option to cache only metadata (deprecated) #4097

cache backend: option to cache only metadata (deprecated) #4097

harryqt commented Mar 29, 2020 •

edited by ivandeex

Loading

Friday13th87 commented Mar 30, 2020

ncw commented Mar 31, 2020

harryqt commented Mar 31, 2020 •

edited

Loading

ncw commented Mar 31, 2020

harryqt commented Mar 31, 2020

ncw commented Apr 1, 2020

harryqt commented Apr 1, 2020 •

edited

Loading

scottmlew commented Apr 6, 2020 •

edited

Loading

harryqt commented May 25, 2020

ncw commented May 25, 2020

harryqt commented May 29, 2020

ivandeex commented Nov 15, 2020

ivandeex commented Mar 12, 2021

harryqt commented Mar 12, 2021

ivandeex commented Mar 12, 2021

ivandeex commented Mar 12, 2021

ivandeex commented Mar 13, 2021

ivandeex commented Mar 13, 2021 •

edited

Loading

harryqt commented Mar 13, 2021

harryqt commented Mar 13, 2021

ivandeex commented Mar 13, 2021

ivandeex commented Mar 13, 2021

ivandeex commented Mar 13, 2021

ivandeex commented Mar 13, 2021

cache backend: option to cache only metadata (deprecated) #4097

cache backend: option to cache only metadata (deprecated) #4097

Comments

harryqt commented Mar 29, 2020 • edited by ivandeex Loading

Update:

Friday13th87 commented Mar 30, 2020

ncw commented Mar 31, 2020

harryqt commented Mar 31, 2020 • edited Loading

ncw commented Mar 31, 2020

harryqt commented Mar 31, 2020

ncw commented Apr 1, 2020

harryqt commented Apr 1, 2020 • edited Loading

scottmlew commented Apr 6, 2020 • edited Loading

harryqt commented May 25, 2020

ncw commented May 25, 2020

harryqt commented May 29, 2020

ivandeex commented Nov 15, 2020

ivandeex commented Mar 12, 2021

harryqt commented Mar 12, 2021

ivandeex commented Mar 12, 2021

ivandeex commented Mar 12, 2021

ivandeex commented Mar 13, 2021

ivandeex commented Mar 13, 2021 • edited Loading

harryqt commented Mar 13, 2021

harryqt commented Mar 13, 2021

ivandeex commented Mar 13, 2021

ivandeex commented Mar 13, 2021

ivandeex commented Mar 13, 2021

ivandeex commented Mar 13, 2021

harryqt commented Mar 29, 2020 •

edited by ivandeex

Loading

harryqt commented Mar 31, 2020 •

edited

Loading

harryqt commented Apr 1, 2020 •

edited

Loading

scottmlew commented Apr 6, 2020 •

edited

Loading

ivandeex commented Mar 13, 2021 •

edited

Loading