Unable to build Python out-of-tree when source tree is readonly. #60023
Comments
|
This is somewhat related to Issue bpo-9860. You can't currently build Python out-of-tree when the source directory is readonly. This is because there are a few Parser/pgen/AST/python_asdl.py steps that try and write to $(srcdir). The attached patch fixes this. |
|
Instead of using "mkdir -p", it should use $(MKDIR_P) (which actually isn't normally "mkdir -p" :-) Other than that, the patch looks fine. |
|
Hmmmmm. Now I'm confused. Is this what you meant? ------------------- % hg diff
diff -r 019a2390b014 Makefile.pre.in
--- a/Makefile.pre.in Tue Aug 21 23:41:43 2012 +0000
+++ b/Makefile.pre.in Thu Aug 30 07:16:55 2012 +0000
@@ -43,6 +43,8 @@
GNULD= @GNULD@
+MKDIR_P= @MKDIR_P@
+
# Shell used by make (some versions default to the login shell, which is bad)
SHELL= /bin/sh
@@ -223,8 +225,8 @@
##########################################################################
# Grammar
-GRAMMAR_H= $(srcdir)/Include/graminit.h
-GRAMMAR_C= $(srcdir)/Python/graminit.c
+GRAMMAR_H= Include/graminit.h
+GRAMMAR_C= Python/graminit.c
GRAMMAR_INPUT= $(srcdir)/Grammar/Grammar
@@ -266,9 +268,9 @@
##########################################################################
# AST
-AST_H_DIR= $(srcdir)/Include
+AST_H_DIR= Include
AST_H= $(AST_H_DIR)/Python-ast.h
-AST_C_DIR= $(srcdir)/Python
+AST_C_DIR= Python
AST_C= $(AST_C_DIR)/Python-ast.c
AST_ASDL= $(srcdir)/Parser/Python.asdl
@@ -605,9 +607,11 @@
Parser/pgenmain.o: $(srcdir)/Include/parsetok.h
$(AST_H): $(AST_ASDL) $(ASDLGEN_FILES)
+ $(MKDIR_P) $(AST_H_DIR)
$(ASDLGEN) -h $(AST_H_DIR) $(AST_ASDL)
$(AST_C): $(AST_ASDL) $(ASDLGEN_FILES)
+ $(MKDIR_P) $(AST_C_DIR)
$(ASDLGEN) -c $(AST_C_DIR) $(AST_ASDL)
Python/compile.o Python/symtable.o Python/ast.o: $(GRAMMAR_H) $(AST_H)
diff -r 019a2390b014 configure
--- a/configure Tue Aug 21 23:41:43 2012 +0000
+++ b/configure Thu Aug 30 07:16:55 2012 +0000
@@ -658,6 +658,7 @@
GNULD
LINKCC
LDVERSION
+MKDIR_P
RUNSHARED
INSTSONAME
LDLIBRARYDIR
@@ -5262,6 +5263,7 @@
HGBRANCH=""
fi
+MKDIR_P="mkdir -p"
DISABLE_ASDLGEN=""
# Extract the first word of "python", so it can be a program name with args.Given your "which actually isn't normally mkdir -p" comment... my patch doesn't seem right. |
|
Am 30.08.12 09:19, schrieb Trent Nelson:
No. Makefile.pre.in *already* has a definition of MKDIR_P, no need to |
loewis
mannequin
changed the title
|
That's what I figured you meant initially... Until I couldn't find any MKDIR_P definitions, and couldn't get it working without the hack above. I thought it might be an implicit make variable as well, but, not so much. (Tested with BSD make and gmake on FreeBSD.) |
|
In 3.3, Makefile.pre.in already defines MKDIR_P, in 3.2 it doesn't. |
|
Ah, I see the MKDIR_P changes in 3.x/configure as well. Objections to backporting to 3.2? Is 2.7 off the table? (Building from a readonly source is handy for Snakebite; saves having a dozen machines needing to use their own hg clone.) |
|
Am 30.08.12 09:47, schrieb Trent Nelson:
No, adding AC_PROG_MKDIR_P into all of them is fine. |
|
New changeset 62044cd5b19b by Trent Nelson in branch '3.2': |
|
New changeset af36536988fd by Trent Nelson in branch 'default': |
|
New changeset ab6ab44921b2 by Trent Nelson in branch 'default': |
|
New changeset 2dde5a7439fd by Trent Nelson in branch '2.7': |
|
I tried building from out-of-tree with a read-only srcdir, but I found two other files which the build process tries to create in the source directory: |
|
What I liked about Trent's original bug report is that it didn't just bring up a generic issue (cannot build with read-only source code), but clarified that his concern really was graminit. So I'd like to declare that this (graminit) is the focus of this issue: any other aspects of read-only source trees are out of scope, and should be reported separately. That said: As for typeslots.inc: this shouldn't really be an issue for a released version of Python, since its tarball should include the generated file, and the timestamps should be in an order so that generation is unnecessary. This should be the case for all generated files that are also checked into Mercurial. For checkouts from Mercurial, you should do "make touch" after an update, as this will bring the timestamps in the right order. It seems that graminit is missing from .hgtouch (in which case Trent's patch probably wouldn't have been necessary, except that it also fixes older branches). But that is again a different issue. As for sysconfigdata: this really should be a separate issue. It's more difficult, since installation and packaging needs to pick up the file even if it's not in the source tree. |
|
As a further follow-up: the original code (in 67ed8a6905c3, by Neil Schemenauer from 2001), he explicitly changed GRAMMAR_C/H to refer to srcdir, claiming that errors from building them would be ignored. In r87558, for which I can't find a subversion revsision for, Victor Stinner dropped the Makefile "-" marker, making the read-only case fail. In reviewing the patch more, I think it is actually incorrect: it now generates the files into Include, but they aren't considered there since (builddir)/Include is not added to the -I options. So the two options now are: Reopening for consideration. |
|
Reconsidering more: it would be desirable if even an out-of-source-tree compilation wouldn't require Python for bootstrapping; with the committed patch, it does since it will always run ASDLGEN. If the source version of the generated files is fresh enough, they should just be copied to the destination. I'm not quite sure how to achieve that with Make rules. This probably *is* a separate issue (which would become irrelevant if the patch here is reverted). |
|
Well, that escalated quickly :-) I think I should clarify my use case that resulted in raising this bug.
So, my patches address that use case. The thing I like about the approach above is that I can set the source directory to readonly immediately -- I don't have to do an intermediate ./configure in that directory in order to generate a Makefile so I can Richard: what was your use case? What steps did you take that resulted in getting typeslots.inc and _sysconfigdata.py written to? ....although I just tried an out-of-tree build with 'default' using the exact steps above and got the same error with _sysconfigdata.py. Martin: I think I understand your arguments, and I (think) I agree with them. Especially your last point with regards to always running ASDLGEN. I think I'm +1 on reverting and trying the '-' approach originally used by Neil. |
|
Am 31.08.12 15:19, schrieb Trent Nelson:
The use case is really uncontended, and long-standing. People want
I guess pretty much the same (though perhaps not ZFS).
Touch Include/typeslots.h (before making the checkout readonly) You were lucky that the timestamps happened to be in an order were
Please don't. I think Victor dropped the - because it would mask I think the regular case should be that the time stamps are in Since the committed patch is already a step in that direction, |
Building on the main host and a VM without having to commit and synchronize temporary changes. (The VM has read-only access to the host's repositories).
As Martin says it is probably just the timestamps which caused typeslots.inc to be rebuilt. In 3.3 _sysconfigdata.py is rebuilt whenever the python binary is. I suspect a bigger issue is the fact that *.pyc files cannot be written to $(srcdir)/Lib/pycache. This means that to complete the build, $(PYTHON_FOR_BUILD) should probably include the -B flag to prevent it from trying to write *.pyc files to a read-only location. But that still leaves you with a python which only works with the -B flag. Maybe library files could be linked/copied to some writable directory early in sys.path. |
|
Am 31.08.12 16:40, schrieb Richard Oudkerk:
Why is that? python will work just fine if it can't write pyc files.
Just for running setup.py? I don't see the need for that. |
When I hacked around the _sysconfigdata.py issue, running the created python produced Fatal Python error: Py_Initialize: Unable to get the locale encoding
Traceback (most recent call last):
File "/mnt/hgfs/Repos/cpython-dirty/Lib/encodings/__init__.py", line 31, in <module>
File "<frozen importlib._bootstrap>", line 1562, in _find_and_load
File "<frozen importlib._bootstrap>", line 1529, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 590, in _check_name_wrapper
File "<frozen importlib._bootstrap>", line 1027, in load_module
File "<frozen importlib._bootstrap>", line 1008, in load_module
File "<frozen importlib._bootstrap>", line 566, in module_for_loader_wrapper
File "<frozen importlib._bootstrap>", line 858, in _load_module
File "<frozen importlib._bootstrap>", line 994, in get_code
File "<frozen importlib._bootstrap>", line 1055, in _cache_bytecode
File "<frozen importlib._bootstrap>", line 1078, in set_data
File "<frozen importlib._bootstrap>", line 128, in _write_atomic
OSError: [Errno 5] Input/output error: '/mnt/hgfs/Repos/cpython-dirty/Lib/__pycache__/codecs.cpython-33.pyc.3076609376'
AbortedI took this to be a failure to write the bytecompiled file. (A regression?) Running "./python -B" does work correctly for me. |
|
It looks like the code which writes the byte-compiled codes checks for and ignores PermissionError (EACCES, EPERM) and FileExistsError (EEXIST). However, for me it is EIO which is raised. |
|
Am 31.08.12 17:32, schrieb Richard Oudkerk:
Sounds like a regression to me. However, errno 5 is EIO. You might want to check whether this really is a regression; |
|
Doesn't EIO cover pathologies which we don't want to silence? I could see adding EROFS though. |
|
Am 31.08.12 18:18, schrieb Eric Snow:
[This should *really* be its own issue] If we get EIO in the middle of writing, this would be worse. |
|
I have opened a new issue: It is a regression because in Python 3.2 all failures to open the file for writing were supressed.
It is VMware Player 4.0.0. |
Martin: I'll work on another patch over the next few days and post it here for review. It seems like your comments regarding our options are still valid (despite this thread getting sidetracked): continue with the premise that all generated files get generated in the build directory, and make sure that build directory gets used during the build (i.e. -I./Include). I'll start with that and see how I get on. |
|
New changeset 617591e7d708 by Trent Nelson in branch '3.2': New changeset abb00e23681a by Trent Nelson in branch '3.3': New changeset 3420fbd87646 by Trent Nelson in branch 'default': |
|
New changeset e0a2b14a3cf9 by Trent Nelson in branch '2.7': |
|
Things should be looking a lot better now across the board. I just committed fixes where appropriate to 2.7, 3.2, 3.3 and 3.x for this issue and two related issues: http://bugs.python.org/issue15298 The only thing that stands out now for 3.[23x] is % hg diff
diff -r 617591e7d708 Makefile.pre.in
--- a/Makefile.pre.in Tue Oct 16 08:41:32 2012 -0400
+++ b/Makefile.pre.in Tue Oct 16 09:46:53 2012 -0400
@@ -947,7 +947,7 @@
multiprocessing multiprocessing/dummy \
unittest unittest/test \
curses pydoc_data $(MACHDEPS)
-libinstall: build_all $(srcdir)/Lib/$(PLATDIR) $(srcdir)/Modules/xxmodule.c
+libinstall: build_all Lib/$(PLATDIR) $(srcdir)/Modules/xxmodule.c
@for i in $(SCRIPTDIR) $(LIBDEST); \
do \
if test ! -d $(DESTDIR)$$i; then \
@@ -1031,14 +1031,14 @@
./$(BUILDPYTHON) -m lib2to3.pgen2.driver $(DESTDIR)$(LIBDEST)/lib2to3/PatternGrammar.txt
# Create the PLATDIR source directory, if one wasn't distributed..
-$(srcdir)/Lib/$(PLATDIR):
- mkdir $(srcdir)/Lib/$(PLATDIR)
- cp $(srcdir)/Lib/plat-generic/regen $(srcdir)/Lib/$(PLATDIR)/regen
+Lib/$(PLATDIR):
+ mkdir Lib/$(PLATDIR)
+ cp $(srcdir)/Lib/plat-generic/regen Lib/$(PLATDIR)/regen
export PATH; PATH="`pwd`:$$PATH"; \
export PYTHONPATH; PYTHONPATH="`pwd`/Lib"; \
export DYLD_FRAMEWORK_PATH; DYLD_FRAMEWORK_PATH="`pwd`"; \
export EXE; EXE="$(BUILDEXE)"; \
- cd $(srcdir)/Lib/$(PLATDIR); $(RUNSHARED) ./regen
+ cd Lib/$(PLATDIR); $(RUNSHARED) ./regen
python-config: $(srcdir)/Misc/python-config.in
# Substitution happens here, as the completely-expanded BINDIR |
|
New changeset c35db4960d1c by Trent Nelson in branch '2.7': New changeset 5707c8a36cc0 by Trent Nelson in branch '3.2': New changeset 85863c4a93db by Trent Nelson in branch '3.3': New changeset d282018bff10 by Trent Nelson in branch 'default': |
|
New changeset 7e879a453bb3 by Trent Nelson in branch '2.7': New changeset 65b3c41052b6 by Trent Nelson in branch '3.2': New changeset b78e97b84f35 by Trent Nelson in branch '3.3': New changeset f9eb516bea88 by Trent Nelson in branch 'default': |
|
just saw the comment about .hgtouch in http://bugs.python.org/issue15819#msg169484 attaching a patch for it here (as proposed on python-committers). |
|
To update .hgtouch is not enough. |
|
I'm not sure how I missed this, but our FreeBSD ports that build out-of-tree required adding a local patchthat reverts the Makefile.pre.in change in ab6ab44921b2 when we updated each port to the latest releases of 2.7, 3.2 and 3.3: https://svnweb.freebsd.org/ports?view=revision&revision=318353 Without it, builds fail with: cc -c -fno-strict-aliasing -O2 -pipe -fno-strict-aliasing -DNDEBUG -I. -IInclude -I./../Include -I/usr/local/include -fPIC -DPy_BUILD_CORE -o Python/Python-ast.o Python/Python-ast.c Stop in /freebsd/ports/lang/python27/work/Python-2.7.6/portbld.shared. I also note Trents msg173160 in issue bpo-15298, which may be related?, but seems only resolved for OSX. What needs to be done to remove the need for this patch? |
|
Perhaps there should be a comment explaining that you can’t write to Also, Trent’s last change, revision 65b3c41052b6, does not seem to work. Apparently $abs_srcdir etc are only valid when substituted in a makefile, and are not valid in the configure script: <https://lists.gnu.org/archive/html/autoconf/2010-07/msg00021.html\>. I will try to post a patch that addresses this to bpo-28066. |
|
@martin Maybe Zach (nosey'd) can help us get an out-of-tree builder set up? If this is relevant for 3.5+, versions should be updated accordingly. |
|
I’m not sure what the overall status of this bug is, so I will leave the versions as they are. Are the three comments from 2013 relevant, and is there anything I can do to help? The Python 2 code already unconditionally adds -IInclude (see bpo-786737), and does not use BASECPPFLAGS. I think it would better to remove Trent’s code, as in unused-flags.patch. Python 3 also has this unconditional -IInclude, so maybe we can eliminate one of them in Python 3? gcc -pthread -c -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Werror=declaration-after-statement -IObjects -IInclude -IPython -I. -IInclude -I../Include -DPy_BUILD_CORE -o Python/Python-ast.o Python/Python-ast.c |
|
Matthias seems to have already applied his patch in Python 3.3 and 3.4+: revision f2cc3d8b88bb. Roumen: Is your problem still relevant? If so, perhaps open a separate bug to elaborate. Koobs: Is your problem with finding Python/Python-ast.c still relevant? Is the |
|
@martin tldr; That was three years ago so I don't know if the issue remains. Perhaps a new out of tree builder will help answer that question. We(FreeBSD) gave up^W switched away from out of tree builds due to lack of support getting issues fixed and the frequency at which commits would alternately break in tree builds then out of tree builds. Unfortunately that also lost us the flexibility to create dual static/shared builds. The relevant downstream commit log: https://svnweb.freebsd.org/ports?view=revision&revision=350610 The patch we added (reverting 2dde5a7439fd) that was mentioned in msg203348 was: If the current code is the same as per the 2dde5a7439fd change, then I'd guess it would still be an issue. |
|
New changeset 9cabcb4411ac by Martin Panter in branch '2.7': |
|
New changeset 36550e4f9b4c by Martin Panter in branch '3.5': New changeset a90daae58323 by Martin Panter in branch '3.6': New changeset 2ac00ff072b8 by Martin Panter in branch 'default': |
|
Just to clarify, the current status is that the revision that Koobs identified is still in place, meaning that the AST files are regenerated into the build tree, not the source tree. I don’t think there is much else to do for this bug, unless Koobs can provide more info why the Free BSD build could not find Python/Python-ast.c, or Roumen can get back with more info. Closing for the moment, but I am happy to help if I can resolve any problems that remain. |
Misc/NEWSso that it is managed by towncrier #552Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: