a non-regular file inside a filesystem directory magic-folder is pointed at prevents contents from being uploaded

[exarkun]

If magic-folder is added pointed at a directory which itself contains something that is not a "regular" file - eg a directory or a symlink - then the contents of the directory are not uploaded. The following appears repeatedly in the log:

af63034d-2bce-43e6-8bed-07748160ebbd
└── scanner:find-updates/1 ⇒ started 2021-07-28 19:13:08Z ⧖ 0.002s
    └── scanner:find-updates/2 ⇒ failed 2021-07-28 19:13:08Z
        ├── exception: exceptions.UnboundLocalError
        └── reason: local variable 'path_state' referenced before assignment

05a802a8-75d0-48d4-b9e2-c3d96c369d0a
└── eliot:traceback/1 2021-07-28 19:13:08Z
    ├── exception: exceptions.UnboundLocalError
    ├── reason: local variable 'path_state' referenced before assignment
    └── traceback: Traceback: <type 'exceptions.UnboundLocalError'>: local variable 'path_state' referenced before assignment⏎
        /nix/store/j0q2p45gpj4w8p6zaixa3574a8c4lip4-python-2.7.17-env/lib/python2.7/site-packages/twisted/internet/defer.py:501:errback⏎                                                                                                                                      
        /nix/store/j0q2p45gpj4w8p6zaixa3574a8c4lip4-python-2.7.17-env/lib/python2.7/site-packages/twisted/internet/defer.py:568:_startRunCallbacks⏎                                                                                                                           
        /nix/store/j0q2p45gpj4w8p6zaixa3574a8c4lip4-python-2.7.17-env/lib/python2.7/site-packages/twisted/internet/defer.py:654:_runCallbacks⏎                                                                                                                                
        /nix/store/j0q2p45gpj4w8p6zaixa3574a8c4lip4-python-2.7.17-env/lib/python2.7/site-packages/twisted/internet/defer.py:1475:gotResult⏎                                                                                                                                   
        --- <exception caught here> ---⏎
        /nix/store/j0q2p45gpj4w8p6zaixa3574a8c4lip4-python-2.7.17-env/lib/python2.7/site-packages/twisted/internet/defer.py:1416:_inlineCallbacks⏎                                                                                                                            
        /nix/store/j0q2p45gpj4w8p6zaixa3574a8c4lip4-python-2.7.17-env/lib/python2.7/site-packages/twisted/python/failure.py:491:throwExceptionIntoGenerator⏎                                                                                                                  
        /nix/store/j0q2p45gpj4w8p6zaixa3574a8c4lip4-python-2.7.17-env/lib/python2.7/site-packages/eliot/_generators.py:141:wrapper⏎
        /nix/store/j0q2p45gpj4w8p6zaixa3574a8c4lip4-python-2.7.17-env/lib/python2.7/site-packages/magic_folder/scanner.py:112:_scan⏎
        /nix/store/j0q2p45gpj4w8p6zaixa3574a8c4lip4-python-2.7.17-env/lib/python2.7/site-packages/eliot/_generators.py:160:wrapper⏎
        /nix/store/j0q2p45gpj4w8p6zaixa3574a8c4lip4-python-2.7.17-env/lib/python2.7/site-packages/twisted/internet/task.py:517:_oneWorkUnit⏎                                                                                                                                  
        /nix/store/j0q2p45gpj4w8p6zaixa3574a8c4lip4-python-2.7.17-env/lib/python2.7/site-packages/magic_folder/scanner.py:142:_process⏎
        /nix/store/j0q2p45gpj4w8p6zaixa3574a8c4lip4-python-2.7.17-env/lib/python2.7/site-packages/magic_folder/util/file.py:70:get_pathinfo⏎                                                                                                                                  

The expected behavior is:

• recursive handling will be applied to directories
• other supported filetypes are uploaded in a way that allows the downloader to recreate them
• unsupported filetypes at least do not get in the way of processing of all supported filetypes

[tomprince]

Do you have an example of it failing on a directory?

[exarkun]

No, I only reached that conclusion by source code inspection.

[meejah]

I tried the following with an existing magic-folder:

• mkdir subdir
• mv catphoto* subdir

I then see that some Snapshots with names like subdir@_catphoto appear (which is approximately what I expect). Can you say what you did to see the above error?

[tomprince]

We'll run into issues if there is something like a symlink (or a unix socket, etc). There are two issues:

• it is isn't meaningful to take a snapshot of them
• the call to get_pathinfo will fail on things that aren't regular files (which includes directories), because those code paths were never tests. It turns out that we were not calling on directories in the scanner, because we had a path.isdir() check first; I think @exarkun assumed they'd cause an issue looking at just get_pathinfo.

I have fix for both issues (which as I've written it) depends on #520 and #521.

[meejah]

Okay, but where does the unbound-local error come from then?

[tomprince]

In https://github.com/LeastAuthority/magic-folder/blob/db128062a435bf622ed6b2d99487c6bec4f64ad7/src/magic_folder/util/file.py#L59-L71 path_state is only set for regular files.