Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel build problem with introspection #106

Closed
pterjan opened this issue Mar 1, 2017 · 7 comments
Closed

Parallel build problem with introspection #106

pterjan opened this issue Mar 1, 2017 · 7 comments

Comments

@pterjan
Copy link

pterjan commented Mar 1, 2017

We got a build failure where it tried generating the typelib from the gir before the gir was finished:

[ 64%] Generating AppStream-1.0.gir
cd /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/build/src && /usr/bin/g-ir-scanner --identifier-prefix=As --symbol-prefix=as --warn-all --add-include-path=/home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src --namespace=AppStream --nsversion=1.0 --no-libtool --library=appstream --include=GObject-2.0 --include=Gio-2.0 --pkg-export=appstream --c-include appstream.h --cflags-begin -I/home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/ -DAS_COMPILATION --cflags-end /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-utils.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-xmldata.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-yamldata.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-desktop-entry.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-cache-file.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-distro-extras.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-stemmer.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-spdx.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-metadata.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-component.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-enums.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-provided.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-bundle.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-pool.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-category.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-distro-details.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-screenshot.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-image.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-release.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-checksum.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-validator.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-validator-issue.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-icon.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-translation.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-suggested.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/appstream.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-utils.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-spdx.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-metadata.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-component.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-pool.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-enums.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-provided.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-bundle.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-category.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-distro-details.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-screenshot.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-image.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-release.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-checksum.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-validator.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-validator-issue.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-icon.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-translation.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-suggested.h --output /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/build/src/AppStream-1.0.gir
make[2]: Leaving directory '/home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/build'
make[2]: Entering directory '/home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/build'
[ 89%] Generating AppStream-1.0.typelib
cd /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src && /usr/bin/g-ir-compiler --includedir=/home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src --includedir=. /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/build/src/AppStream-1.0.gir -o /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/build/src/AppStream-1.0.typelib
error parsing file /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/build/src/AppStream-1.0.gir: Error on line 1442 char 31: Document ended unexpectedly while inside an attribute value
src/CMakeFiles/gir-typelibs.dir/build.make:63: recipe for target 'src/AppStream-1.0.typelib' failed
make[2]: *** [src/AppStream-1.0.typelib] Error 1
make[2]: Leaving directory '/home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/build'
CMakeFiles/Makefile2:198: recipe for target 'src/CMakeFiles/gir-typelibs.dir/all' failed
make[1]: *** [src/CMakeFiles/gir-typelibs.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
make[2]: Entering directory '/home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/build'
[ 64%] Generating AppStream-1.0.gir
cd /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/build/src && /usr/bin/g-ir-scanner --identifier-prefix=As --symbol-prefix=as --warn-all --add-include-path=/home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src --namespace=AppStream --nsversion=1.0 --no-libtool --library=appstream --include=GObject-2.0 --include=Gio-2.0 --pkg-export=appstream --c-include appstream.h --cflags-begin -I/home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/ -DAS_COMPILATION --cflags-end /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-utils.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-xmldata.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-yamldata.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-desktop-entry.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-cache-file.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-distro-extras.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-stemmer.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-spdx.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-metadata.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-component.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-enums.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-provided.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-bundle.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-pool.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-category.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-distro-details.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-screenshot.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-image.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-release.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-checksum.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-validator.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-validator-issue.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-icon.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-translation.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-suggested.c /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/appstream.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-utils.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-spdx.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-metadata.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-component.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-pool.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-enums.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-provided.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-bundle.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-category.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-distro-details.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-screenshot.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-image.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-release.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-checksum.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-validator.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-validator-issue.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-icon.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-translation.h /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/src/as-suggested.h --output /home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/build/src/AppStream-1.0.gir
make[2]: Leaving directory '/home/pterjan/rpmbuild/BUILD/AppStream-0.10.6/build'
@pterjan
Copy link
Author

pterjan commented Mar 1, 2017

Full build log build.0.20170301091110.txt

@ximion
Copy link
Owner

ximion commented Mar 5, 2017

Unfortunately can't reproduce this problem (built from scratch with make -j9 three times) - likely having an SSD prevents this bug from ever showing up...
(Unfortunate for bug-hunting)

@ximion
Copy link
Owner

ximion commented Mar 5, 2017

Hmm, can you maybe try whether this trivial changes help?

--- a/data/cmake/GObjectIntrospectionMacros.cmake
+++ b/data/cmake/GObjectIntrospectionMacros.cmake
@@ -81,7 +81,7 @@ macro(gir_add_introspections introspections_girs)
               --includedir=.
               ${CMAKE_CURRENT_BINARY_DIR}/${gir}
               -o ${CMAKE_CURRENT_BINARY_DIR}/${_typelib}
-      DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${gir}
+      DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${gir} ${gir}
       OUTPUT ${_typelib}
       WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
     )

The dependencies look correct, but maybe by using an absolute path, cmake only check whether the file exists and not whether the previous command did actually finish to run.

@pterjan
Copy link
Author

pterjan commented Mar 8, 2017

Reading https://samthursfield.wordpress.com/2015/11/21/cmake-dependencies-between-targets-and-files-and-custom-commands/#custom-commands-and-parallel-make it seems you need to depend both on the file and a custom_target
According to that post if you only depend on the file, the scanner may get run several times which explains the truncated file as it was generated once but is getting overwritten concurrently because gir-girs also depends on it.

@pterjan
Copy link
Author

pterjan commented Mar 8, 2017

maybe something like:

@@ -72,6 +72,8 @@
       VERBATIM
     )
     list(APPEND _gir_girs ${CMAKE_CURRENT_BINARY_DIR}/${gir})
+    add_custom_target("gir-${gir}" ALL DEPENDS ${gir})
+    list(APPEND _gir_girs "gir-${gir}")
     install(FILES ${CMAKE_CURRENT_BINARY_DIR}/${gir} DESTINATION ${CMAKE_INSTALL_DATAROOTDIR}/gir-1.0)
 
     string(REPLACE ".gir" ".typelib" _typelib "${gir}")
@@ -81,7 +83,7 @@
               --includedir=.
               ${CMAKE_CURRENT_BINARY_DIR}/${gir}
               -o ${CMAKE_CURRENT_BINARY_DIR}/${_typelib}
-      DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${gir}
+      DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${gir} "gir-${gir}"
       OUTPUT ${_typelib}
       WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}
     )

Unfortunately I have failed to reproduce it since that one time it happened :(

@ximion
Copy link
Owner

ximion commented Mar 8, 2017

The code in question is used by a lot of people and projects, including things like e.g. Poppler, so I find it quite surprising that it fails at all, given the amount of testing it has received.
Maybe adding gir-girs to https://github.com/ximion/appstream/blob/master/data/cmake/GObjectIntrospectionMacros.cmake#L94 already does the job.
Unfortunately I can't test anything atm.

@ximion ximion closed this as completed May 7, 2017
@ximion
Copy link
Owner

ximion commented May 7, 2017

Please reopen this in case it happens again (but it's likely I will use Meson for AppStream soonish anyway)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants