test_relative_path of test_py_compile fails on macOS 10.15 Catalina

[warsaw]

BPO	38295
Nosy	@warsaw, @terryjreedy, @ronaldoussoren, @ned-deily, @maggyero, @miss-islington, @Bo98, @amgedr, @iritkatriel
PRs	[3.8] bpo-38295: Prevent failure of test_relative_path in test_py_compile on macOS Catalina #17636 [3.7] bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636) #17637 [3.6] bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636) #17638 [2.7] bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636) #17639

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-01-30.16:52:21.030>
created_at = <Date 2019-09-27.17:24:03.947>
labels = ['OS-mac', 'type-bug', '3.8', '3.9', '3.7', 'tests']
title = 'test_relative_path of test_py_compile fails on macOS 10.15 Catalina'
updated_at = <Date 2022-01-31.00:00:22.784>
user = 'https://github.com/warsaw'

bugs.python.org fields:

activity = <Date 2022-01-31.00:00:22.784>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2022-01-30.16:52:21.030>
closer = 'iritkatriel'
components = ['macOS', 'Tests']
creation = <Date 2019-09-27.17:24:03.947>
creator = 'barry'
dependencies = []
files = []
hgrepos = []
issue_num = 38295
keywords = ['patch']
message_count = 15.0
messages = ['353381', '353408', '354022', '357483', '358535', '358536', '358537', '358538', '358539', '358541', '358546', '358550', '358651', '410941', '412180']
nosy_count = 9.0
nosy_names = ['barry', 'terry.reedy', 'ronaldoussoren', 'ned.deily', 'maggyero', 'miss-islington', 'Bo98', 'amgedr', 'iritkatriel']
pr_nums = ['17636', '17637', '17638', '17639']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue38295'
versions = ['Python 2.7', 'Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']

[warsaw]

test_py_compile fails on macOS Catalina beta (19A573a)

======================================================================
ERROR: test_relative_path (test.test_py_compile.PyCompileTestsWithSourceEpoch)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/bwarsaw/projects/python/3.8/Lib/test/test_py_compile.py", line 30, in wrapper
    return fxn(*args, **kwargs)
  File "/Users/bwarsaw/projects/python/3.8/Lib/test/test_py_compile.py", line 111, in test_relative_path
    py_compile.compile(os.path.relpath(self.source_path),
  File "/Users/bwarsaw/projects/python/3.8/Lib/py_compile.py", line 157, in compile
    os.makedirs(dirname)
  File "/Users/bwarsaw/projects/python/3.8/Lib/os.py", line 221, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '../../../../../../../var/folders/w6/w_2zrjgj7bgdgkhdjm4_r9s4000slb/T/tmpq3p6aoly'

======================================================================
ERROR: test_relative_path (test.test_py_compile.PyCompileTestsWithoutSourceEpoch)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Users/bwarsaw/projects/python/3.8/Lib/test/test_py_compile.py", line 20, in wrapper
    return fxn(*args, **kwargs)
  File "/Users/bwarsaw/projects/python/3.8/Lib/test/test_py_compile.py", line 111, in test_relative_path
    py_compile.compile(os.path.relpath(self.source_path),
  File "/Users/bwarsaw/projects/python/3.8/Lib/py_compile.py", line 157, in compile
    os.makedirs(dirname)
  File "/Users/bwarsaw/projects/python/3.8/Lib/os.py", line 221, in makedirs
    mkdir(name, mode)
FileNotFoundError: [Errno 2] No such file or directory: '../../../../../../../var/folders/w6/w_2zrjgj7bgdgkhdjm4_r9s4000slb/T/tmpquor0q8r'

---

[terryjreedy]

This is not a duplicate because Python *did* compile enough to fail a test.

[ned-deily]

Also the similar single test in 2.7.x fails on 10.15 (pre-release):

======================================================================
ERROR: test_relative_path (test.test_py_compile.PyCompileTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/test/test_py_compile.py", line 45, in test_relative_path
    os.path.relpath(self.pyc_path))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/py_compile.py", line 123, in compile
    with open(cfile, 'wb') as fc:
IOError: [Errno 2] No such file or directory: '../../../../../../../var/folders/n6/q3475zjd38bc3q197d77wjc80000gn/T/tmpLPxkJN/_test.pyc'

[Bo98]

For what it's worth, this is having an impact on some real code: Homebrew/homebrew-core#45110

Perhaps a simpler way to reproduce is:

% cd /tmp
% python3 -c 'import os; open(os.path.relpath("/tmp/test.txt"), "w")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
FileNotFoundError: [Errno 2] No such file or directory: '../../tmp/test.txt'

[ned-deily]

New changeset bf3aa10 by Ned Deily in branch '3.8':
bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636)
bf3aa10

[miss-islington]

New changeset 5f2c134 by Miss Islington (bot) in branch '2.7':
bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636)
5f2c134

[miss-islington]

New changeset 3dbfe0a by Miss Islington (bot) in branch '3.7':
bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636)
3dbfe0a

[ned-deily]

New changeset 5a9b697 by Ned Deily (Miss Islington (bot)) in branch '3.6':
bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636) (GH-17638)
5a9b697

[ned-deily]

For master and 3.9.0a2, equivalent workaround was included in a76ba36 (bpo-39041 / #61794).

While these changes prevent the specific test failure, they point to more general issues with use of relative paths on macOS particularly with temp files and with 10.15 Catalina. More a bit later ...

[Bo98]

Indeed. The issue can be trivially reproduced with:

#include <fcntl.h>
#include <errno.h>
#include <stdio.h>
#include <unistd.h>

int main()
{
  char buf[255];
  printf("Current dir: %s\n", getcwd(buf, 255));

  int fd = open("../../tmp/test.txt", O_WRONLY | O_CREAT);
  if (fd < 0)
  {
    printf("errno %d\n", errno);
    return 1;
  }
  close(fd);
  printf("Success\n");
  return 0;
}

and running it in /private/tmp.

I filed FB7467762 at the end of November. Downstream projects meanwhile are working around the issue by resolving the file path before passing it into open.

[ronaldoussoren]

You don't even need a C program to reproduce:

$ cd /private/tmp
$ touch ../../foo.txt
touch: ../../tmp/foo.txt: No such file or directory

And the "cd" is optional, I get the same error from my home directory (/Users/ronald).

All of this on macOS 10.15.2

[Bo98]

You don't even need a C program to reproduce

Indeed, touch is built upon the POSIX file API (unless Apple modified it). The idea for the Apple bug report was to show it happening at a low level and not specific to a given tool.

And the "cd" is optional, I get the same error from my home directory

Yes, /private/tmp is just an example but I'd be cautious saying the same happens everywhere. You won't be able to reproduce the issue if your current directory is /usr.

/private/tmp, your home directory, etc. are all "firmlinked" to /System/Volumes/Data in Catalina. /System/Volumes/Data/private/tmp exists but /System/Volumes/Data/tmp doesn't exist. This shouldn't really be an issue as the idea of firmlinks is to make the /System/Volumes/Data invisible and thus you should be able to relatively go back up to the /System/Volumes/Data and be transported back to the root directory, where you can find the /tmp symlink (and indeed that works fine with ls). Evidently that doesn't seem to work properly for file operations.

[ned-deily]

New changeset 13ee023 by Ned Deily (Miss Islington (bot)) in branch '3.7':
bpo-38295: prevent test_relative_path of test_py_compile failure on macOS Catalina (GH-17636)
13ee023

[iritkatriel]

Is there anything more to do here?

[ned-deily]

Is there anything more to do here?

A belated response: probably not, so I'm OK with closing this.

A quick check shows that macOS behavior has changed a bit on more recent releases. On current macOS Big Sur (11.6.3) and Monterey (12.2) releases, Bo's test:

$ python3 -c 'import os; open(os.path.relpath("/tmp/test.txt"), "w")'

now works and Ronald's touch example now fails with a somewhat more meaningful message:

$ touch ../../foo.txt
touch: ../../foo.txt: Read-only file system

In general, though, even on pre-APFS older macOS systems, it is still best to avoid or at least be very careful about using /tmp with any sort of relative path testing because of the linking of /tmp to /private/tmp. :(