ignored folders on Windows

[remusb]

What is your rclone version (eg output from rclone -V)
cache beta

Which OS you are using and how many bits (eg Windows 7, 64 bit)
Windows

The command you were trying to run (eg rclone copy /tmp remote:tmp)
mount

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2017/12/21 02:49:08 DEBUG : : Re-reading directory (5m4.4962215s old)
2017/12/21 02:49:08 DEBUG : : list: warm 225 from cache for: , ts: 2017-12-21 02:44:04.2506533 -0800 PST
2017/12/21 02:49:08 ERROR : TV Shows/xxxxxxxxx: Entry doesn't belong in directory "" (contains subdir) - ignoring
2017/12/21 02:49:08 ERROR : TV Shows/xxxxxxxxx: Entry doesn't belong in directory "" (contains subdir) - ignoring

I'm moving the issue in another one cause I can't seem to replicate a problem with this one on Windows and will need additional investigation. Plus, #1904 was related to another issue related to missing files caused by the old method of caching folders.

@floriansofianos I was able to replicate the same log lines in my mount but I don't see them being a problem in this case.

Here's my result:

2017/12/22 03:01:35 DEBUG : : list: warm 13 from cache for: , ts: 2017-12-22 02:58:44.6671079 -0800 PST
2017/12/22 03:01:35 ERROR : cache: '1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' became '1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' from ''-'1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' with sep '\'
2017/12/22 03:01:35 ERROR : cache: '1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' became '1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' from ''-'1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir1' became 'dir1' from ''-'dir1' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir1' became 'dir1' from ''-'dir1' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir1/subdir1' became 'dir1/subdir1' from 'dir1'-'subdir1' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir1/subdir1' became 'dir1/subdir1' from 'dir1'-'subdir1' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir1/subdir1' became 'dir1/subdir1' from 'dir1'-'subdir1' with sep '\'
2017/12/22 03:01:35 ERROR : dir1/subdir1: Entry doesn't belong in directory "" (contains subdir) - ignoring
2017/12/22 03:01:35 ERROR : cache: 'dir2' became 'dir2' from ''-'dir2' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir2' became 'dir2' from ''-'dir2' with sep '\'

Now, I was listing the root folder here: / and I can't be sure why dir1/subdir1 would be returned here and I'll take a better look at it but it is valid to be ignored cause subdir1 isn't an entry of the root. Furthermore while going in Explorer in dir1, my listing was correct and got subdir1 back.

Are your folders still missing with the latest beta and you're just getting this log or are they actually missing if you look for them?

[floriansofianos]

Hello @remusb , sorry for this very late answer!

With the latest beta, I have a cache on a root directory (encrypted) that contains 6 folders. If I use windows explorer to navigate to one of those 6 folders that contains files and directories. I can at first see everything correctly.
But when I wait 5 minutes, if I go to windows explorer again and browse the same directory, all the directories inside it are missing. All the files that aren't in a directory are still there though.
The same thing happens if I try to use the command line to browse the directory: I'm missing all my inner directories.

The folder I use for testing is quite big: it contains 190 folders which probably have subfolders as well and a lot of files.

In the logs, the only issue I can see is those "Entry doesn't belong in directory "" (contains subdir) - ignoring"

Also I'm not doing anything special during the 5 minutes of waiting.

[diggerydoo]

I can confirm the above issue happens to me too. All the folders and files show up perfectly on the decrypted cache mount initially. However, after a certain amount of time, the subfolders simply disappear. Remounting both the cache mount and the decrypted mount (that points to the cache mount) still show those folders as gone (when they are actually still present on Google Drive).

I used both rclone 1.39 and the latest beta (rclone v1.39-015-g28480c05β) on Windows 10.

[drgreatscott]

Same here. rclone cache ran on a folder with 8 sub-folders. Each sub-folder has hundreds of files nested under them. Not using crypt.

When the cache drive just mounted, everything looked fine. After a while, all the subfolders went invisible from the mounted drive, while the files are actually still on GD as could be seen from the GD web interface.

I'm using Windows 10, rclone 1.39 AMD64 version of rclone. But I got 403 forbidden error from GD before/during that time.

Later I mounted the drive without cache and it went on for 10 hours without missing subfolders, while I still got 403 forbidden error.

Afraid that my continuing not using cache will result in 403 forbidden error again, I mounted the drive with cache again. This time I made --dir-cache-time=24h as a workaround, not sure if this could help; the subfolders haven't disappeared yet after an hour or so. Any idea if this is a valid workaround, or there's any better workaround?

Update: After setting --dir-cache-time=24h, I didn't encounter the problem for a day... But after a day the folders still vanished. Now I'm setting --dir-cache-time=240h until a better workaround or fix is available :-)

[remusb]

The entry does not belong in directory issue should have gone away in the latest beta.
Can you give it a try if you're still interested? https://beta.rclone.org/v1.39-099-gb3d8b7e2/

Thanks

[drgreatscott]

I just tried the latest (February 11th) AMD64 beta. At first there were many I/O errors for my programs, possibly due to my fault of not cleaning up the old config or cache. Then I cleaned up all cache and restarted the program. It has been running smoothly for the past 12 hours. No missing files/folders until now.

I encourage everyone who encountered this bug to give it a try! Thank you so much @remusb !

I'm using Windows 10. --dir-cache-time and --cache-info-age both set to 3 hours now. Running rclone mount on GD->cache->crypt.

[remusb]

Awesome. So the issue can be pinned on the entry doesn't belong in directory log which was fixed by handling \ even on Windows as path separator.

I'm closing the issue but if you encounter anything else, feel free to open a new one.

Thank you