2to3 Grammar pickles not created when upgrading to 3.3.0b2

[stefanholek]

BPO	15645
Nosy	@terryjreedy, @ronaldoussoren, @benjaminp, @ned-deily, @regebro, @asvetlov, @hynek
Files	issue15645.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-08-23.06:41:25.318>
created_at = <Date 2012-08-14.08:16:19.821>
labels = ['build', 'expert-2to3']
title = '2to3 Grammar pickles not created when upgrading to 3.3.0b2'
updated_at = <Date 2012-09-09.02:48:16.356>
user = 'https://bugs.python.org/stefanholek'

bugs.python.org fields:

activity = <Date 2012-09-09.02:48:16.356>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2012-08-23.06:41:25.318>
closer = 'ned.deily'
components = ['Build', '2to3 (2.x to 3.x conversion tool)']
creation = <Date 2012-08-14.08:16:19.821>
creator = 'stefanholek'
dependencies = []
files = ['26916']
hgrepos = []
issue_num = 15645
keywords = ['patch', 'needs review']
message_count = 19.0
messages = ['168180', '168465', '168480', '168483', '168484', '168629', '168677', '168678', '168679', '168703', '168737', '168858', '168924', '168926', '169678', '169679', '169817', '170025', '170074']
nosy_count = 10.0
nosy_names = ['terry.reedy', 'ronaldoussoren', 'benjamin.peterson', 'ned.deily', 'lregebro', 'Arfrever', 'asvetlov', 'stefanholek', 'python-dev', 'hynek']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue15645'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

[stefanholek]

After upgrading 3.3.0b1 to b2 (make install to same location) I noticed that the Grammar pickles for beta.2 were missing (the beta.1 pickles still being there).

The result is permission errors and/or virtualenv SandboxViolations, when 2to3 is used for the first time. E.g.:

SandboxViolation: open('/usr/local/python3.3/lib/python3.3/lib2to3/Grammar3.3.0.beta.2.pickle', 'wb') {}

This is because users of Python (as opposed to root) may of course not write to the library dir.

The pickles are created fine on a fresh install of Python 3.3.0b2 (make install to new location).

[terryjreedy]

Your OS and distribution/version might be relevant information.

[stefanholek]

Mac OS X 10.6.8 (Snow Leopard)
Xcode Tools 3.2.6

./configure --prefix=/usr/local/python3.3
make
sudo make install

[terryjreedy]

I believe there are known problems with 3.3 and mac, so I added the mac experts.

[ned-deily]

I took a quick look at the issue and I could reproduce by doing a "make install" of 3.3.0b1 then a build and "make install" of 3.3.0b2 in the same locations. Doing a clean 3.3.0b2 build and install produce the expected results. It appears to be a unix Makefile issue that shouldn't be unique to OS X. I don't have time to look further into it right at the moment, though. (And I'm not sure what "known problems with 3.3 and mac" means; the buildbots are happy.)

[ronaldoussoren]

I'm pretty sure this is a generic Makefile bug. I could reproduce this problem with a clean install into a new prefix, after cleaning up the source tree: 'make clean' won't remove the generated file when srcdir != builddir.

This line in the Makefile seems to generate the pickle file:

    -PYTHONPATH=$(DESTDIR)$(LIBDEST) $(RUNSHARED) \
            $(PYTHON_FOR_BUILD) -Wi -c "import lib2to3.pygram,lib2to3.patcomp;lib2to3.patcomp.PatternCompiler()"

This uses $(PYTHON_FOR_BUILD) to generate the cache file, and that which means the cache file is generated in the *source* tree, not in the installation tree. This command is at the end of the libinstall target, which means the generatd file is not copied into the installation prefix.

Running 'make install' twice will therefore install the file.

The attached patch seems to fix the issue by moving the command to the start of the libinstall target. The patch also fixes "make clean", by explicitly targetting the pickle files in $(srcdir)/Lib/lib2to3.

BTW. What "known OSX problems"?

BTW2. This should affect all platforms that use Makefiles (basicly anything but Windows)

[terryjreedy]

I had the impression that there *were* problems with compiling 3.3 on the new OSX and Xcode or something. If that is wrong or all fixed, great. Sorry for the noise.

[ronaldoussoren]

There were some problems building using llvm-gcc, those have been resolved by detecting falling back to clang when llvm-gcc is detected. That's the only real problem I know of.

[ronaldoussoren]

BTW. I wouldn't mind a patch review, particularly by someone that can easily test on Linux. I'm not too keen on checking in this patch without a test on Linux, not this close to 3.3rc1.

[terryjreedy]

If necessary, ask on pydev.

[ned-deily]

Patch LGTM and I verified that it does indeed install the pickles properly on both a vanilla Linux build/install and with an OS X framework build/install. (Note that, as expected, previously installed versioned pickles in the install library directory are not removed. They will harmlessly accumulate until you do a clean install or manually remove them.)

[regebro]

I can confirm that this is an issue on Ubuntu 12.04 as well, and that removing the existing pre 3.3b2 install before installing solves the problem.