Getting correct PATH locations on Windows 10 PC in uproot.open(<pathname>)

[physicscitizen]

I manged to get 64 bit Python 3.8 loaded up and it seems to work ok on some of my simple files.
Using pip I was able to get numpy, uproot, and awkward installed as well with no problems.
My working disk is my “D:” drive, but my ATLAS data sits on my “F:” drive.

So I naively decided to try out uproot first just within an IDLE session.

And I’m falling at the first hurdle because it seems that uproot is having trouble parsing the path to the ROOT file.

Here’s what I get in the IDLE session:

Python 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.
>>> import uproot as upr
>>> import awkward
>>> pth = 'F:\\ROOTdataATLAS\\ATLASatk4EMTopoHITS002866.root'
>>> print(pth)
F:\ROOTdataATLAS\ATLASatk4EMTopoHITS002866.root

>>> upr.open(pth)

Traceback (most recent call last):
  File "<pyshell#4>", line 1, in <module>
    upr.open(pth)
  File "D:\Python38\lib\site-packages\uproot\rootio.py", line 53, in open
    return ROOTDirectory.read(openfcn(path), **options)
  File "D:\Python38\lib\site-packages\uproot\rootio.py", line 50, in <lambda>
    openfcn = lambda path: MemmapSource(path, **kwargs)
  File "D:\Python38\lib\site-packages\uproot\source\memmap.py", line 21, in __init__
    self._source = numpy.memmap(self.path, dtype=numpy.uint8, mode="r")
  File "D:\Python38\lib\site-packages\numpy\core\memmap.py", line 225, in __new__
    f_ctx = open(os_fspath(filename), ('r' if mode == 'c' else mode)+'b')
FileNotFoundError: [Errno 2] No such file or directory: '\\ROOTdataATLAS\\ATLASatk4EMTopoHITS002866.root'
>>>

The first lines are me just creating a string variable ‘pth’ that holds the path and file location and then printing out that variable to verify that it is correct (it is).
It appears that uproot is picking up the double back-slashes and not interpreting them as single back-slashes? Also, will uproot understand the change of disk location?

[physicscitizen]

Oh, I should point out that I did not have any trouble when I tried the identical sequence on a linux machine.

[jpivarski]

Aha! I bet it's backslashes in strings in Python. See this?

>>> pth = 'F:\\ROOTdataATLAS\\ATLASatk4EMTopoHITS002866.root'
>>> print(pth)

F:\ROOTdataATLAS\ATLASatk4EMTopoHITS002866.root

In the printout, there's only one slash because it interpreted "\\" as r"\". Try putting an r at the beginning of the string:

>>> pth = r'F:\\ROOTdataATLAS\\ATLASatk4EMTopoHITS002866.root'
>>> print(pth)

F:\\ROOTdataATLAS\\ATLASatk4EMTopoHITS002866.root

With the r, it doesn't interpret the backslash as an "escape" character, so you get the two backslashes after the F:. I can't check in Windows to see that two backslashes are absolutely required here, but I suspect that they are.

Does that help?

[eduardo-rodrigues]

Hi, I also work on Windows (Py 3.7) and here is the 2 forms that work just fine for me:
either uproot.open(r"D:\home\...') or uproot.open("D:\\home\\...').

[physicscitizen]

Thanks for the suggestions folks, but neither of these actually work. And I'm baffled because your solutions seem really sensible to me! Here's what I get. First just an explaination. At first I'm trying Jim's idea where I alter my "pth" string to be literal. In the second case I try Eduardo's first suggestion (his second suggestion I know doesn't work because I'd done that before I tried using a string variable in the first place).

Python 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.
>>> import uproot as upr
>>> import awkward
>>> pth = r'F:\\ROOTdataATLAS\\ATLASatk4EMTopoHITS002866.root'
>>> print(pth)
F:\\ROOTdataATLAS\\ATLASatk4EMTopoHITS002866.root
>>> upr.open(pth)
Traceback (most recent call last):
  File "<pyshell#4>", line 1, in <module>
    upr.open(pth)
  File "D:\Python38\lib\site-packages\uproot\rootio.py", line 53, in open
    return ROOTDirectory.read(openfcn(path), **options)
  File "D:\Python38\lib\site-packages\uproot\rootio.py", line 50, in <lambda>
    openfcn = lambda path: MemmapSource(path, **kwargs)
  File "D:\Python38\lib\site-packages\uproot\source\memmap.py", line 21, in __init__
    self._source = numpy.memmap(self.path, dtype=numpy.uint8, mode="r")
  File "D:\Python38\lib\site-packages\numpy\core\memmap.py", line 225, in __new__
    f_ctx = open(os_fspath(filename), ('r' if mode == 'c' else mode)+'b')
FileNotFoundError: [Errno 2] No such file or directory: '\\\\ROOTdataATLAS\\\\ATLASatk4EMTopoHITS002866.root'
>>> upr.open(r'F:\ROOTdataATLAS\ATLASatk4EMTopoHITS002866.root')
Traceback (most recent call last):
  File "<pyshell#5>", line 1, in <module>
    upr.open(r'F:\ROOTdataATLAS\ATLASatk4EMTopoHITS002866.root')
  File "D:\Python38\lib\site-packages\uproot\rootio.py", line 53, in open
    return ROOTDirectory.read(openfcn(path), **options)
  File "D:\Python38\lib\site-packages\uproot\rootio.py", line 50, in <lambda>
    openfcn = lambda path: MemmapSource(path, **kwargs)
  File "D:\Python38\lib\site-packages\uproot\source\memmap.py", line 21, in __init__
    self._source = numpy.memmap(self.path, dtype=numpy.uint8, mode="r")
  File "D:\Python38\lib\site-packages\numpy\core\memmap.py", line 225, in __new__
    f_ctx = open(os_fspath(filename), ('r' if mode == 'c' else mode)+'b')
FileNotFoundError: [Errno 2] No such file or directory: '\\ROOTdataATLAS\\ATLASatk4EMTopoHITS002866.root'
>>> 

[physicscitizen]

If this works under Python 3.7 and NOT Python 3.8 could it be a bug actually within Python itself?

[physicscitizen]

After some playing around I've concluded that this problem is certainly entirely associated with how uproot is parsing pathnames in Windows. If I change the directory in IDLE to be the same as the directory of the file it works. Do be more explicit:

>>> import os
>>> os.chdir('F:\\ROOTdataATLAS')

is sufficient for 'uproot.open('myFile.root') ' to then work as long as 'myFile.root' is in that directory on that disk. I'm able to see the TTree inside the file and pick up all the branches in it with the 'keys' and 'values' keywords. This probably means that uproot is getting into the file the way it should. It really is only the parsing of those annoying Windows path-names which use the wrong angle of the slash-key.

[jpivarski]

Uproot first tries to parse the path as a URL and, failing that, a local file. There's explicit logic to catch Windows path names: https://github.com/scikit-hep/uproot/blob/eb5d8862f8a382606be7bb9fa84f169b7cd19513/uproot/rootio.py#L41-L64 but I don't have a Windows test computer to try it out. Can you tell which test is failing? os.name == "nt"? The _windows_absolute path match?

[physicscitizen]

I'm afraid this is past my ability to help unless there is something specific you would like me to try on my PC and the instructions are explicit.

Off the top of my head though, that line 42 might need to be broken up as I'm having a hard time understanding the logic of that 'if' statement even if I replace all the names with generic letters (like 'A', 'B', 'C' excetera) and then try to figure out a truth table. It looks like a pretty dense conditional statement that relies heavily on the operation precidence rules.

[chrisburr]

I think the useful things to show would be the output of:

import os
path = "F:\\ROOTdataATLAS\\ATLASatk4EMTopoHITS002866.root"
print(repr(os.name))
print(repr(open._windows_absolute.match(path)))

[physicscitizen]

Here is what I get:

Python 3.8.0 (tags/v3.8.0:fa919fd, Oct 14 2019, 19:37:50) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license()" for more information.
>>> 
>>> import os
>>> path = "F:\\ROOTdataATLAS\\ATLASatk4EMTopoHITS002866.root"
>>> print(repr(os.name))
'nt'
>>> print(repr(open._windows_absolute.match(path)))
Traceback (most recent call last):
  File "<pyshell#4>", line 1, in <module>
    print(repr(open._windows_absolute.match(path)))
AttributeError: 'builtin_function_or_method' object has no attribute '_windows_absolute'
>>> print(path)
F:\ROOTdataATLAS\ATLASatk4EMTopoHITS002866.root
>>> 

Looks like Winodows 10 doesn't understand 'open._windows_absolute.match() ?

[jpivarski]

Thanks, @chrisburr, I was in the air at the time. The last one to check should be

print(repr(uproot.open._windows_absolute.match(path)))

because the regex is attached to uproot's open, not the Python system's open (which the quoted code doesn't make clear).

(Does anybody know where we can get interactive access to a Windows machine to do this kind of debugging ourselves?)

[chrisburr]

You can get free Windows VMs from here: https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/

I opened mine up and the issue is that the drive letter is dropped by this line: https://github.com/scikit-hep/uproot/blob/eb5d8862f8a382606be7bb9fa84f169b7cd19513/uproot/rootio.py#L43

I don't have clipboard sharing set up so a screenshot will have to do:

[jpivarski]

Let me know how PR #383 fares. It's been a while since I've used VirtualBox, and I'm pretty sure I'll need to get to my main computer to try it (Chromebook probably can't handle it; nor can the phone...).

[chrisburr]

That seems to do the trick. I can't reproduce the full issue as I only have one partition in my VM but if I look at open("C:\....").source.path it is missing C: for master and includes it on that branch.

[jpivarski]

Great! I'll deploy this with a fix to #367 today.

(And thanks also for the link to Windows VMs! It looks like some of them are 32-bit, and I can use that to debug 32-bit Windows issues without having to send a million commits to Azure. Still setting it up, though...)

[physicscitizen]

If I want to try this as a remote user is it possible yet? Can I just repeat the

pip install uproot

and get this new fix and try it out? Or is it more complicated than that?

[jpivarski]

When this deployment finishes, it will be available as uproot 3.10.8. At that time, you can

pip install --upgrade uproot

Just in case you run into any problems, --no-cache-dir tends to fix them. (It's especially good for failed compilations that pip thinks it's finished, but it's in a broken state because the compilation failed. Upgrading uproot shouldn't involve any compilations, though.)

[jpivarski]

Now it's available.

[eduardo-rodrigues]

If I want to try this as a remote user is it possible yet? Can I just repeat the
pip install uproot

and get this new fix and try it out? Or is it more complicated than that?

At times it can be quick to play a simple trick: git clone the repository somewhere, cd to it, and launch python. You will then import things from this local repo and will be able to do trivial checks. I often do this.

[physicscitizen]

That fixed it! Thanks!