Shake cannot build Chrome

[ndmitchell]

Seems like path normalisation, probably passed from here somehow https://code.google.com/p/chromium/codesearch#chromium/src/chrome/tools/build/repack_locales.py&q=locale.pak&sq=package:chromium&type=cs&l=53
and probably called from here somehow https://code.google.com/p/chromium/codesearch#chromium/src/chrome/chrome_repack_locales.gypi&q=repack_locales.py&sq=package:chromium&type=cs&l=19

hummer:src thakis$ time /Users/thakis/Library/Haskell/ghc-7.6.3/lib/shake-0.10.6/bin/shake --directory out/Release   chrome  --report   +RTS -s -I0 -qg -qb
ACTION packed_resources: repack_chrome_resources_200_percent_611cb56cf20bc0f70aef4d2e365d52fc
# webkit_resources_200_percent.pak gen/repack/chrome_200_percent.pak
MACTOOL copy-bundle-resource gen/repack/chrome_200_percent.pak
# chrome_200_percent.pak" Chromium Framework.framework/Versions/A/Resources/chrome_200_percent.pak
ACTION packed_resources: repack_locales_611cb56cf20bc0f70aef4d2e365d52fc
# . am ar bg bn ca cs da de el en-GB en-US es-419 es et fa fi fil fr gu he hi hr hu id it ja kn ko lt lv ml mr ms nb nl pl pt-BR pt-PT ro ru sk sl sr sv sw ta te th tr uk vi zh-CN zh-TW gen/repack/am.lproj/locale.pak gen/repack/ar.lproj/locale.pak gen/repack/bg.lproj/locale.pak gen/repack/bn.lproj/locale.pak gen/repack/ca.lproj/locale.pak gen/repack/cs.lproj/locale.pak gen/repack/da.lproj/locale.pak gen/repack/de.lproj/locale.pak gen/repack/el.lproj/locale.pak gen/repack/en_GB.lproj/locale.pak gen/repack/en.lproj/locale.pak gen/repack/es_419.lproj/locale.pak gen/repack/es.lproj/locale.pak gen/repack/et.lproj/locale.pak gen/repack/fa.lproj/locale.pak gen/repack/fi.lproj/locale.pak gen/repack/fil.lproj/locale.pak gen/repack/fr.lproj/locale.pak gen/repack/gu.lproj/locale.pak gen/repack/he.lproj/locale.pak gen/repack/hi.lproj/locale.pak gen/repack/hr.lproj/locale.pak gen/repack/hu.lproj/locale.pak gen/repack/id.lproj/locale.pak gen/repack/it.lproj/locale.pak gen/repack/ja.lproj/locale.pak gen/repack/kn.lproj/locale.pak gen/repack/ko.lproj/locale.pak gen/repack/lt.lproj/locale.pak gen/repack/lv.lproj/locale.pak gen/repack/ml.lproj/locale.pak gen/repack/mr.lproj/locale.pak gen/repack/ms.lproj/locale.pak gen/repack/nb.lproj/locale.pak gen/repack/nl.lproj/locale.pak gen/repack/pl.lproj/locale.pak gen/repack/pt_BR.lproj/locale.pak gen/repack/pt_PT.lproj/locale.pak gen/repack/ro.lproj/locale.pak gen/repack/ru.lproj/locale.pak gen/repack/sk.lproj/locale.pak gen/repack/sl.lproj/locale.pak gen/repack/sr.lproj/locale.pak gen/repack/sv.lproj/locale.pak gen/repack/sw.lproj/locale.pak gen/repack/ta.lproj/locale.pak gen/repack/te.lproj/locale.pak gen/repack/th.lproj/locale.pak gen/repack/tr.lproj/locale.pak gen/repack/uk.lproj/locale.pak gen/repack/vi.lproj/locale.pak gen/repack/zh_CN.lproj/locale.pak gen/repack/zh_TW.lproj/locale.pak
Traceback (most recent call last):
  File "tools/build/repack_locales.py", line 216, in <module>
    results = DoMain(sys.argv[1:])
  File "tools/build/repack_locales.py", line 213, in DoMain
    return repack_locales(locales)
  File "tools/build/repack_locales.py", line 146, in repack_locales
    data_pack.DataPack.RePack(output, inputs)
  File "tools/build/../../../tools/grit/grit/format/data_pack.py", line 140, in RePack
    WriteDataPack(resources, output_file, encoding)
  File "tools/build/../../../tools/grit/grit/format/data_pack.py", line 110, in WriteDataPack
    with open(output_file, "wb") as file:
IOError: [Errno 2] No such file or directory: '../out/Release/gen/./repack/ar.lproj/locale.pak'
shake: Error when running Shake build system:
* Chromium.app
* Chromium.app/Contents/MacOS/Chromium
* Chromium Framework.framework
* Chromium Framework.framework/Versions/A/Resources/zh_TW.lproj/locale.pak
* gen/repack/zh_TW.lproj/locale.pak
* gen/repack/am.lproj/locale.pak gen/repack/ar.lproj/locale.pak gen/repack/bg.lproj/locale.pak gen/repack/bn.lproj/locale.pak gen/repack/ca.lproj/locale.pak gen/repack/cs.lproj/locale.pak gen/repack/da.lproj/locale.pak gen/repack/de.lproj/locale.pak gen/repack/el.lproj/locale.pak gen/repack/en_GB.lproj/locale.pak gen/repack/en.lproj/locale.pak gen/repack/es_419.lproj/locale.pak gen/repack/es.lproj/locale.pak gen/repack/et.lproj/locale.pak gen/repack/fa.lproj/locale.pak gen/repack/fi.lproj/locale.pak gen/repack/fil.lproj/locale.pak gen/repack/fr.lproj/locale.pak gen/repack/gu.lproj/locale.pak gen/repack/he.lproj/locale.pak gen/repack/hi.lproj/locale.pak gen/repack/hr.lproj/locale.pak gen/repack/hu.lproj/locale.pak gen/repack/id.lproj/locale.pak gen/repack/it.lproj/locale.pak gen/repack/ja.lproj/locale.pak gen/repack/kn.lproj/locale.pak gen/repack/ko.lproj/locale.pak gen/repack/lt.lproj/locale.pak gen/repack/lv.lproj/locale.pak gen/repack/ml.lproj/locale.pak gen/repack/mr.lproj/locale.pak gen/repack/ms.lproj/locale.pak gen/repack/nb.lproj/locale.pak gen/repack/nl.lproj/locale.pak gen/repack/pl.lproj/locale.pak gen/repack/pt_BR.lproj/locale.pak gen/repack/pt_PT.lproj/locale.pak gen/repack/ro.lproj/locale.pak gen/repack/ru.lproj/locale.pak gen/repack/sk.lproj/locale.pak gen/repack/sl.lproj/locale.pak gen/repack/sr.lproj/locale.pak gen/repack/sv.lproj/locale.pak gen/repack/sw.lproj/locale.pak gen/repack/ta.lproj/locale.pak gen/repack/te.lproj/locale.pak gen/repack/th.lproj/locale.pak gen/repack/tr.lproj/locale.pak gen/repack/uk.lproj/locale.pak gen/repack/vi.lproj/locale.pak gen/repack/zh_CN.lproj/locale.pak gen/repack/zh_TW.lproj/locale.pak
Development.Shake.command_, system command failed
Command: cd ../../chrome; export BUILT_PRODUCTS_DIR=/Volumes/MacintoshHD2/src/chrome-git/src/out/Release; export CONFIGURATION=Release; export PRODUCT_NAME=packed_resources; export SDKROOT=/Applications/Xcode5-DP2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.8.sdk; export SRCROOT=/Volumes/MacintoshHD2/src/chrome-git/src/out/Release/../../chrome; export SOURCE_ROOT="${SRCROOT}"; export TARGET_BUILD_DIR=/Volumes/MacintoshHD2/src/chrome-git/src/out/Release; export TEMP_DIR="${TMPDIR}";python tools/build/repack_locales.py -b chromium -p mac -g ../out/Release/gen/chrome -s ../out/Release/gen -x ../out/Release/gen/. am ar bg bn ca cs da de el en-GB en-US es-419 es et fa fi fil fr gu he hi hr hu id it ja kn ko lt lv ml mr ms nb nl pl pt-BR pt-PT ro ru sk sl sr sv sw ta te th tr uk vi zh-CN zh-TW
Exit code: 1
Stderr:
Traceback (most recent call last):
  File "tools/build/repack_locales.py", line 216, in <module>
    results = DoMain(sys.argv[1:])
  File "tools/build/repack_locales.py", line 213, in DoMain
    return repack_locales(locales)
  File "tools/build/repack_locales.py", line 146, in repack_locales
    data_pack.DataPack.RePack(output, inputs)
  File "tools/build/../../../tools/grit/grit/format/data_pack.py", line 140, in RePack
    WriteDataPack(resources, output_file, encoding)
  File "tools/build/../../../tools/grit/grit/format/data_pack.py", line 110, in WriteDataPack
    with open(output_file, "wb") as file:
IOError: [Errno 2] No such file or directory: '../out/Release/gen/./repack/ar.lproj/locale.pak'

[nico]

Trunk shake was able to build the 'chrome' target with this fix. Now trying all other targets (tests and whatnot).

…that failed almost immediately with

# video_quality_analysis.video_quality_analysis.o obj/third_party/webrtc/tools/frame_analyzer/video_quality_analysis.video_quality_analysis.o
shake: Error when running Shake build system:
* unit_tests
* obj/skia/ext/unit_tests.lazy_pixel_ref_utils_unittest.o
* obj/chrome/unit_tests.actions_rules_copies.stamp
* test_data/ui/webui/resources/js/cr.js
Error, file does not exist and no rule available:
  test_data/ui/webui/resources/js/cr.js

[nico]

out/Release/obj/chrome/unit_tests.ninja has a rule for test_data/chrome/../ui/webui/resources/js/cr.js , so this is probably just due to shake not doing path normalization (which also makes https://codereview.chromium.org/18227004/ necessary). Yes, gyp could be smarter about this too.

[ndmitchell]

I now do full path normalisation on all build rules (before I just removed a leading ./), and since I already did full normalisation on dependencies that should be enough.

I'm not convinced that given Ninja's stated goals that Ninja should be doing any path normalisation, but since it already does, I now do so. I have used the simple but obviously correct implementation, so this may reduce the parsing speed, but if it does, I'll catch it when I next do a round of profiling.

[ndmitchell]

As part of #25 I rewrite file normalisation, and also did a lot of profiling. This should make Chrome building Ninja noticably faster, although it is still likely to have excessive memory (see #28). Let me know if there are any remaining problems building Chrome.

[nico]

I'll give it another try, thanks!