New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ignored folders on Windows #1935

Closed
remusb opened this Issue Dec 22, 2017 · 6 comments

Comments

Projects
None yet
4 participants
@remusb
Collaborator

remusb commented Dec 22, 2017

What is your rclone version (eg output from rclone -V)
cache beta

Which OS you are using and how many bits (eg Windows 7, 64 bit)
Windows

The command you were trying to run (eg rclone copy /tmp remote:tmp)
mount

A log from the command with the -vv flag (eg output from rclone -vv copy /tmp remote:tmp)

2017/12/21 02:49:08 DEBUG : : Re-reading directory (5m4.4962215s old)
2017/12/21 02:49:08 DEBUG : : list: warm 225 from cache for: , ts: 2017-12-21 02:44:04.2506533 -0800 PST
2017/12/21 02:49:08 ERROR : TV Shows/xxxxxxxxx: Entry doesn't belong in directory "" (contains subdir) - ignoring
2017/12/21 02:49:08 ERROR : TV Shows/xxxxxxxxx: Entry doesn't belong in directory "" (contains subdir) - ignoring

I'm moving the issue in another one cause I can't seem to replicate a problem with this one on Windows and will need additional investigation. Plus, #1904 was related to another issue related to missing files caused by the old method of caching folders.

@floriansofianos I was able to replicate the same log lines in my mount but I don't see them being a problem in this case.

Here's my result:

2017/12/22 03:01:35 DEBUG : : list: warm 13 from cache for: , ts: 2017-12-22 02:58:44.6671079 -0800 PST
2017/12/22 03:01:35 ERROR : cache: '1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' became '1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' from ''-'1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' with sep '\'
2017/12/22 03:01:35 ERROR : cache: '1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' became '1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' from ''-'1j9g1s2bcjbls1n8t26k3vs4gihipl4g9f5tjsv1koghmt984rrt5ikaktlekdkms11t7609h1adsqnkptkgpnampl1dgpjo7jfjvi8' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir1' became 'dir1' from ''-'dir1' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir1' became 'dir1' from ''-'dir1' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir1/subdir1' became 'dir1/subdir1' from 'dir1'-'subdir1' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir1/subdir1' became 'dir1/subdir1' from 'dir1'-'subdir1' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir1/subdir1' became 'dir1/subdir1' from 'dir1'-'subdir1' with sep '\'
2017/12/22 03:01:35 ERROR : dir1/subdir1: Entry doesn't belong in directory "" (contains subdir) - ignoring
2017/12/22 03:01:35 ERROR : cache: 'dir2' became 'dir2' from ''-'dir2' with sep '\'
2017/12/22 03:01:35 ERROR : cache: 'dir2' became 'dir2' from ''-'dir2' with sep '\'

Now, I was listing the root folder here: / and I can't be sure why dir1/subdir1 would be returned here and I'll take a better look at it but it is valid to be ignored cause subdir1 isn't an entry of the root. Furthermore while going in Explorer in dir1, my listing was correct and got subdir1 back.

Are your folders still missing with the latest beta and you're just getting this log or are they actually missing if you look for them?

@floriansofianos

This comment has been minimized.

floriansofianos commented Dec 30, 2017

Hello @remusb , sorry for this very late answer!

With the latest beta, I have a cache on a root directory (encrypted) that contains 6 folders. If I use windows explorer to navigate to one of those 6 folders that contains files and directories. I can at first see everything correctly.
But when I wait 5 minutes, if I go to windows explorer again and browse the same directory, all the directories inside it are missing. All the files that aren't in a directory are still there though.
The same thing happens if I try to use the command line to browse the directory: I'm missing all my inner directories.

The folder I use for testing is quite big: it contains 190 folders which probably have subfolders as well and a lot of files.

In the logs, the only issue I can see is those "Entry doesn't belong in directory "" (contains subdir) - ignoring"

Also I'm not doing anything special during the 5 minutes of waiting.

@diggerydoo

This comment has been minimized.

diggerydoo commented Jan 7, 2018

I can confirm the above issue happens to me too. All the folders and files show up perfectly on the decrypted cache mount initially. However, after a certain amount of time, the subfolders simply disappear. Remounting both the cache mount and the decrypted mount (that points to the cache mount) still show those folders as gone (when they are actually still present on Google Drive).

I used both rclone 1.39 and the latest beta (rclone v1.39-015-g28480c05β) on Windows 10.

@drgreatscott

This comment has been minimized.

drgreatscott commented Jan 23, 2018

Same here. rclone cache ran on a folder with 8 sub-folders. Each sub-folder has hundreds of files nested under them. Not using crypt.

When the cache drive just mounted, everything looked fine. After a while, all the subfolders went invisible from the mounted drive, while the files are actually still on GD as could be seen from the GD web interface.

I'm using Windows 10, rclone 1.39 AMD64 version of rclone. But I got 403 forbidden error from GD before/during that time.

Later I mounted the drive without cache and it went on for 10 hours without missing subfolders, while I still got 403 forbidden error.

Afraid that my continuing not using cache will result in 403 forbidden error again, I mounted the drive with cache again. This time I made --dir-cache-time=24h as a workaround, not sure if this could help; the subfolders haven't disappeared yet after an hour or so. Any idea if this is a valid workaround, or there's any better workaround?

Update: After setting --dir-cache-time=24h, I didn't encounter the problem for a day... But after a day the folders still vanished. Now I'm setting --dir-cache-time=240h until a better workaround or fix is available :-)

@remusb

This comment has been minimized.

Collaborator

remusb commented Feb 7, 2018

The entry does not belong in directory issue should have gone away in the latest beta.
Can you give it a try if you're still interested? https://beta.rclone.org/v1.39-099-gb3d8b7e2/

Thanks

@drgreatscott

This comment has been minimized.

drgreatscott commented Feb 12, 2018

I just tried the latest (February 11th) AMD64 beta. At first there were many I/O errors for my programs, possibly due to my fault of not cleaning up the old config or cache. Then I cleaned up all cache and restarted the program. It has been running smoothly for the past 12 hours. No missing files/folders until now.

I encourage everyone who encountered this bug to give it a try! Thank you so much @remusb !

I'm using Windows 10. --dir-cache-time and --cache-info-age both set to 3 hours now. Running rclone mount on GD->cache->crypt.

@remusb

This comment has been minimized.

Collaborator

remusb commented Feb 12, 2018

Awesome. So the issue can be pinned on the entry doesn't belong in directory log which was fixed by handling \ even on Windows as path separator.

I'm closing the issue but if you encounter anything else, feel free to open a new one.

Thank you

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment