proposal: os.File: make real file name as on disk accessible

[xformerfhs]

Problem

The os.File Openfunctions always return the name of the file as it was presented to the Open function. I.e. opening a file with f, err := os.Open("huGO.txt") returns "huGO.txt" when f.Name() is called. This is all fine as long as one uses file names on the same operating system. However, it creates a problem, when one uses data from one operating system on another.

Suppose there is an application that calculates some properties for some files and stores the result in a file with key-value pairs where the key is the file name and the value is some property. The file names are supplied by the user. The program has a property generator and a property verifier command.

Now, suppose the program property generator is run on a Windows machine and there is a file called huGO.txt on the disk. The user supplies hugo.txt as the file name. Since the Windows file system is case-insensitive by default the function os.Open("hugo.txt") will open the file huGO.txt. The property generator can only save the file name as presented to the Open function which is the name entered by the user and therefore generates the following output when run on Windows:

"hugo.txt": "...property..."

Now the file and the property generator output are transferred to a Linux machine. There the verifier is run. The verifier reads the file names from the property file. So it will try to open a file called hugo.txt. However, the file name is huGO.txt. The Linux file system is case-sensitive by default and so the verifier fails to find the file on Linux and will output an error message. The verifier would work on Windows, but not on Linux.

This is an unsolvable problem in the current situation. When one stores file names that the user provided on an OS with a case-insensitive file system and wants to work with these file names on an OS with a case-sensitive file system this will inevitably fail if the user did not specify the file name exactly as it is stored on the file system. On Windows users rely on the fact that Windows is not case-sensitive, so this happens very often.

Solution

This could be solved by a function that returns the "real" file name as stored by the file system. If the file is opened with f, err := os.Open("hugo.txt") on Windows a function fn := f.FsName() would return huGO.txt in the above example.

This proposed os.File function FsName() would solve the above problem quite easily while still maintaining compatibility with older programs. The name of the function could be something else, like e.g. RealName() or whatever. There just should be a function with the specified functionality.

[rittneje]

This is not how filesystems work on Unix. The file (inode) itself has no record of a "true" name, because a file could have multiple "true" names (hard links) or even none at all (e.g., O_TMPFILE).

On Windows specifically you can probably leverage GetFinalPathNameByHandle. https://learn.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-getfinalpathnamebyhandlew (I am not a Windows developer.)

On macOS you can allegedly use F_GETPATH via fcntl but I've never tried. https://developer.apple.com/library/archive/documentation/System/Conceptual/ManPages_iPhoneOS/man2/fcntl.2.html

On Linux you can allegedly read /proc/self/fd/<fd> but there are caveats. https://stackoverflow.com/questions/1188757/retrieve-filename-from-file-descriptor-in-c

[ianlancetaylor]

Unfortunately, this isn't possible in general, so I'm going to close this. Please comment if you disagree.

Note that on Windows you can already call https://pkg.go.dev/golang.org/x/sys/windows#GetFinalPathNameByHandle . But as @rittneje notes there is no possible equivalent on Unix file systems.

[xformerfhs]

Hi, @rittneje and @ianlancetaylor. Thanks for your answers. Indeed, I think this proposal should not be closed, yet :-).

Everything you have written is correct and I have the impression that I did not quite convey my idea. So, please allow me to put it in another way:

The os package deals with two abstract objects: Files and Directories. What I want to propose is a function that tells the caller by which name in the directory the file was found. So, it is not relevant, whether there are multiple ways to reference a file. I want the one by which the file specified in the os.Open call was found. This name obviously must be present in the directory of the os.Open() call.

I have implemented a quick and dirty solution that only uses these Go-supplied abstractions:

type FileFinder struct {
   plainMap        map[string]bool
   ucMap           map[string]string
   isCaseSensitive bool
}

func NewFileFinder(dirName string) (*FileFinder, error) {
   if len(dirName) == 0 {
      dirName = "."
   }

   d, err := os.Open(dirName)
   if err != nil {
      return nil, fmt.Errorf("could not open directory '%s': %v", dirName, err)
   }
   defer FileCloser(d)

   var namesInDir []string
   namesInDir, err = d.Readdirnames(-1)
   if err != nil {
      return nil, fmt.Errorf("could not read names in directory '%s': %v", dirName, err)
   }

   result := &FileFinder{}

   dirNamesLen := len(namesInDir)
   plainMap := make(map[string]bool, dirNamesLen)
   ucMap := make(map[string]string, dirNamesLen)
   isCaseSensitive := false

   var fi os.FileInfo
   for _, nameInDir := range namesInDir {
      fi, err = os.Lstat(filepath.Join(dirName, nameInDir))   
      if err != nil {
         return nil, fmt.Errorf("could not get status of file '%s' in directory '%s': %v", err)
      }

      if !fi.IsDir() {
         plainMap[nameInDir] = true
         ucNameInDir := strings.ToUpper(nameInDir)
         if len(ucMap[ucNameInDir]) == 0 {
            ucMap[ucNameInDir] = nameInDir
         } else {
            isCaseSensitive = true
         }
      }
   }

   if !isCaseSensitive {
      isCaseSensitive = (runtime.GOOS != "windows") && (runtime.GOOS != "darwin")
   }

   result.plainMap = plainMap
   result.ucMap = ucMap
   result.isCaseSensitive = isCaseSensitive

   return result, nil
}

func (ff *FileFinder) GetFsName(fileName string) string {
   if ff.isCaseSensitive {
      if ff.plainMap[fileName] {
         return fileName
      } else {
         return ""
      }
   } else {
      if ff.plainMap[fileName] {
         return fileName
      } else {
         fsName, found := ff.ucMap[strings.ToUpper(fileName)]
            if found {
               return fsName
            } else {   
               return ""   
            }
      }
   }
}

This would be called like that:

...
   ff, err := NewFileFinder(dirName)
   if err != nil {
      return "", fmt.Errorf("could not create file finder: %v", err)
   }

   realName := ff.GetFsName(fileName)
...

Let me explain what this does: It reads all names in a directory that are not sub-directories and stores them in two maps. One where the key is the name as it is found in the directory and one where the key is the the name converted to all upper-case. I know this is not quite correct and in a correct solution this ought to be changed to proper Unicode folding.

After a successful call to os.Open() it is known that the file is present in the directory. When one wants to know the file name as present in the directory one calls FsName(fileName) which first looks up the name in the plain map. If it is found there it was specified as it is recorded in the directory and we are done.

If it is not found by the specified file name the name is converted to all upper-case and looked up in the second map with the all upper-case file names. It will be found here, as it exists and the value of the map is the name as found in the directory.

Everything depends on the case-sensitivity of the file names in the directory. In nearly every OS there may be directories that use case-sensitive names and others that use case-insensitive names. My QAD solution is not correct as currently there is no way to find out the case-sensitivity of a directory. It uses a rather simple approximation: If there are two file names that map to the same upper-case file name this indicates a case-sensitive directory and it assumes that any OS other than Windows and MacOS is case-sensitive. In a proper solution the case-sensitivity of the directory should be found out by OS calls.

I hope this describes my proposal more precisely.

[ianlancetaylor]

OK, reopening.

I don't personally think this should be in the standard library, but I agree that it is approximately possible. (It will still fail if the file is removed, but, OK.)

[rittneje]

@xformerfhs Your code has a fundamental race condition. Suppose I am on a file system that is case sensitive. I have two files "foo.txt" and "Foo.txt". I then call open("foo.txt"), and immediately thereafter delete (unlink) the file. Your logic would incorrectly latch onto "Foo.txt". A similar issue unfolds if the names of two files get swapped.

You would instead need to compare the inode numbers. But even then, it cannot give an answer once the file gets removed or renamed.

I will also point out this only works for the "leaf" file. If you have an arbitrary file path then you need to do this for every directory along the way. My feeling is this is a very niche use case with a lot of edges, and thus does not belong in the standard library.

[xformerfhs]

Thank you @rittneje and @ianlancetaylor for taking the time to discuss this proposal. I really appreciate that.

My quick-and-dirty code was meant as an example to illustrate my point, not as a production-ready program. As I pointed out, there are some character-code issues and, as @rittneje correctly pointed out, there is a possible race condition and it only works on files in a directory.

On Windows the possible race condition is not a problem. The code is meant to be used after a file has been opened. Windows refuses to delete or rename a file that is opened, so the code will work as intended. The directory will be consistent with the open file as long as the file is opened. There will never be a race condition.

On Unix this code does not need to run when the directory uses case-sensitive file names. It would only be used when a directory with case-insensitive file names is used. Only then the race condition could show up as Unix allows the deletion and renaming of a file while it is open. As @rittneje said, one would need to compare inode numbers instead of names and it would even be possible, that there is no longer a directory entry that refers to the inode of the opened file when it has been deleted in the time between the opening of the file and the reading of the directory. One would have to deal with this case in the real code. I do not have the knowledge to do that. In this case it would possibly be sufficient to use the name that was presented to os.Open.

This function is not meant to traverse directories. I want the "real" name of a file and only the file name, not the real name of the directory. One could think about a function that yields the "real" file path. But this is more complicated and not the intention of this proposal.

It is impossible to guarantee the consistency of a directory with an opened file when the directory is not guaranteed to be consistent with an open file as is the case with Unix. This would only be possible if there would be some kind of directory snapshot. However, there is none, so one would have to deal with the possible inconsistencies.

What gives me more of a headache is the fact that one should actually know for each directory whether it uses case-sensitive file names or case-insensitive file names. This depends on the file system and its configuration.

[apparentlymart]

Hi @xformerfhs,

This seems like an example of a problem that does not have a single obvious solution, and whose definition of "correct" depends a lot on subtleties of your particular use-case.

For situations like that, it's typically best to implement in a third-party library first. Then you can get the functionality you need to solve your immediate problem, and you can also publish your library to see if others find it useful. The experience of building a third-party library and testing it against other people's use-cases will often help hone what parts of the design, if any, are general enough needs to be addressed in the standard library.

[bitfield]

I doubt that many people have this problem, and I doubt that there's a reliable enough solution to be worth publishing in the form of a package. I further doubt that it would be used widely enough to be considered for inclusion in the standard library.

But who cares what I think, right? I'm wrong about six important things before breakfast every day. If you want to prove me wrong, take up @apparentlymart's suggestion, implement a solid solution to this in your own package, and add it to what I whimsically call the “universal” library (that is, GitHub).

If it turns out a lot of people have this problem, they will be very glad to have your solution, and I will look like a jackass. So it's win-win.

[xformerfhs]

I now have one platform-specific FsName for Windows and one for Linux, and they are exactly the ones rittneje pointed out to me. These are two small files for which a separate library is not worthwhile. I can add a third file, which is the solution I actually use in my program: a glob that works correctly on Windows. Let's see how and if I publish this.

BTW, wasn't there supposed to be a patch for glob to work case-agnostic on Windows?