backslash in windows path definition - need a forward slash

[ReimarBauer]

Hi
I ran in a problem of \ used in pathes. This was in the past different. Also the documentations shows:

This is broadly similar to the standard os.path module but works with paths in the canonical format expected by all FS objects (that is, separated by forward slashes and with an optional leading slash).

from fs.tempfs import TempFS
ROOT_FS = TempFS()
ROOT_DIR = ROOT_FS.root_path

print(ROOT_DIR)
'C:\\Users\\Local...'

[lurch]

I think root_path is an "internal detail" which you shouldn't be accessing directly? Instead you want to use getsyspath https://docs.pyfilesystem.org/en/latest/concepts.html#system-paths

[ReimarBauer]

Thx, but this makes not the difference.

ROOT_FS.getsyspath('')
Out[7]: 'C:\\Users\\LOCAL_~....`

the linked example creates also C:\\Users\\.. dirs.

[willmcgugan]

@ReimarBauer What is the output you were expecting?

[ReimarBauer]

C:/Users/and/so/on

[willmcgugan]

On Window you wil get backslashes because os.sep is backslash on Windows.

Don't forget that when you repr a string with backslashes, Python will escape them, so it will display as two backslashes...

[ReimarBauer]

ahh, ok that's a good hint. We can close this here. I need only to be very consequent in my joining of dirs.

Mixing backslash and forward slash makes the trouble.

e9ce93b30\/msc

[ReimarBauer]

Seems I get in that problem

e9ce93b30\/msc

because fs.path.join uses also on windows a forward slash.

fs.path.join(ROOT_DIR, 'M1')

[lurch]

because fs.path.join uses also on windows a forward slash.

You need to be very careful to separate "pyfilesystem paths" (which will always use /) and "system paths" (which will use \ on Windows). "pyfilesystem paths" should only be used with the fs.* methods in pyfilesystem, and "system paths" should only be used with Python's os.* methods. Accidentally mixing them up and using the wrong path with the wrong function is a recipe for confusion and/or disaster 😉
https://docs.pyfilesystem.org/en/latest/concepts.html

[lurch]

@willmcgugan @dargueta ISTR that other people have also run into this "confusion" before - perhaps the docs need clarifying about this? Maybe by using an example or two to illustrate the concepts?

[willmcgugan]

@lurch Yeah probably. Maybe a callout in the concepts page would help, and some additional text at the top of the reference.

@ReimarBauer Just to re-iterate, use os.path.join for system paths, use fs.path.join for PyFilesystem paths.

[ReimarBauer]

I agree I must be careful.

On this abstraction I don't want to use or want a dependency on os.path.join. I want be able to exchange pyfilesystem2 urls which uses forward slashes. This should be interpreted on the operation system how it is needed. At a join where a collission on OS level like "\/" happens may be only the trailing slash needs to be removed .

[lurch]

This should be interpreted on the operation system how it is needed.

No it shouldn't, the whole point of PyFilesystem paths is that they always use / regardless of the underlying OS.

At a join where a collission on OS level like "/" happens may be only the trailing slash needs to be removed .

As mentioned above, you shouldn't be trying to "join" a system (OS) path with a PyFilesystem path. I think the fundamental way to look at it is:

1. Do OS-level-stuff with system paths using standard Python library functions
2. Construct a PyFilesystem FS (possibly initialised from a system path)
   1. Do PyFilesystem stuff using PyFilesystem paths and functions
   2. ...
   3. If you must call a standard Python library function on a PyFilesystem path, convert it to a system path using getsyspath
   4. Do more PyFilesystem stuff using PyFilesystem paths and functions
   5. Close the PyFilesystem FS object
3. Do more OS-level-stuff with system paths using standard Python library functions

EDIT:

I want be able to exchange pyfilesystem2 urls which uses forward slashes.

Perhaps https://docs.pyfilesystem.org/en/latest/openers.html is what you're looking for?

[ReimarBauer]

I have now more or less solved it how you suggested.
The problem arises in the test framework. The tests for linux worked but only windows got that problem. The application runs on linux, Mac and windows. And there we use FS Urls.

The tests should be close to the usecases.

Hmm, may be a TempFS Url would be interesting. Not thought on that before.

[ReimarBauer]

from fs.tempfs import TempFS
ROOT_FS = TempFS()

A TempFS is always a osfs filesystem. May be it could be interesting to request the url representation. Than the OS dependant part can be done by the OS and the user can if he wants continue with the url.

[lurch]

Any OS-dependent part should always be done with a syspath, please don't try to bodge an FS-URL into working with OS functions 😉 But of course it's fine to "serialise" a TempFS path to a FS-URL, and to then later recreate an OSFS from that same FS-URL (but be careful of the auto_clean parameter https://docs.pyfilesystem.org/en/latest/reference/tempfs.html#fs.tempfs.TempFS if doing this across different processes).

Note also that an FS-URL may look similar to a "regular" URL, but the two aren't interchangeable, there are subtle differences (there was a big discussion here ages ago about how file:// URLs work on Windows). So again, FS-URLs should only be used with pyfilesystem functions; and standard Python functions (urllib, requests, etc.) should only be used with regular URLs and not with FS-URLs.

[lurch]

The way PyFilesystem works does make sense, once you get used to it 😄

[ReimarBauer]

On writing a config py script like this example I have to replace "\" by "/"
That's my current solution. It is impossible to import this as module on windows. Because "\" will escape the " ' " of the ROOT_DIR string and you get a "SyntaxError: EOL while scanning string literal"

class mscolab_settings(object):

    # SQLALCHEMY_DB_URI = 'mysql://user:pass@127.0.0.1/mscolab'
    import os
    import logging
    import fs
    import secrets
    from werkzeug.urls import url_join

    ROOT_DIR = 'C:\Users\LOCAU\Temp\23\tmpps037so4mss6575af3296\'
    # directory where mss output files are stored
    root_fs = fs.open_fs(ROOT_DIR)

The string to be written looks good

ROOT_DIR = \'C:\\Users\\LOCALU\\Temp\\23\\tmpps037so4mss6575af3296\\\'\n

[willmcgugan]

If I understand correctly, don’t you just need to double up the backslashes so they don’t escape the quote?

[lurch]

Yep, you can either do:

ROOT_DIR = 'C:\\Users\\LOCAU\\Temp\\23\\tmpps037so4mss6575af3296\\'

(standard double-backslash escape)

or you can do:

ROOT_DIR = r'C:\Users\LOCAU\Temp\23\tmpps037so4mss6575af3296'

(slash isn't necessary at the end of the path, but you want a r before the string to mark it as a raw-string, to stop backslash being interpreted as an escape sequence)

See https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals for more info.

As an example of why the r is necessary if you're not using double-backslash:

>>> ROOT_DIR = 'some\\nice\\long\\path'
>>> print(ROOT_DIR)
some\nice\long\path
>>> ROOT_DIR = 'some\nice\long\path'
>>> print(ROOT_DIR)
some
ice\long\path
>>> ROOT_DIR = r'some\nice\long\path'
>>> print(ROOT_DIR)
some\nice\long\path

(\n is the escape-sequence for a newline character)

[ReimarBauer]

If I would manually write that string I would prefer like in the example this version, without the \ at the end.

ROOT_DIR = r'C:\Users\LOCAU\Temp\23\tmpps037so4mss6575af3296'

The string I have gets into the config_file by a f'string substitution from

ROOT_FS = TempFS()
ROOT_DIR = ROOT_FS.getsyspath("")

This has always an ending slash on windows the \.

>>> f'{ROOT_DIR}' 'C:\\Users\\LOCALU\\Temp\\4\\tmp2cns24tj__tempfs__\\'
>>> rf'{ROOT_DIR}' 'C:\\Users\\LOCALU\\Temp\\4\\tmp2cns24tj__tempfs__\\'

This is resulting into a \' and this escape gives the EOL Problem.

My current solution is to replace \ to /.

Maybe just the last os.seperator can be a / in any case, or can be requested.

[lurch]

This has always an ending slash on windows the \

Oh that's strange. Is that deliberate behaviour @willmcgugan ?

My current solution is to replace \ to /.

Probably better to just remove the last character if it's a \?

>>> ROOT_DIR = 'C:\\Users\\LOCAU\\Temp\\23\\tmpps037so4mss6575af3296\\'
>>> print(ROOT_DIR)
C:\Users\LOCAU\Temp\23\tmpps037so4mss6575af3296\
>>> if ROOT_DIR.endswith('\\'):
...     ROOT_DIR = ROOT_DIR[:-1]
... 
>>> print(ROOT_DIR)
C:\Users\LOCAU\Temp\23\tmpps037so4mss6575af3296