Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ICU-20555 Fix Windows data build failure with long paths #776

Merged

Conversation

@jefgen
Copy link
Member

jefgen commented Aug 28, 2019

When building ICU4C from a long folder path on Windows, the data build portion of the build process will fail as the length of the command line passed to genrb.exe exceeds the maximum length supported by CMD, which is 8191 characters.

One possible method of working around this is to modify/hack the Python data build scripts.
However, that has the the unfortunate result of losing some of the parallelism in some cases, as the
RepeatedOrSingleExecutionRequest needs to be changed to a SingleExecutionRequest in order to move the decision about whether or not to pass the the locale listing as an argument list or as a file to genrb.exe earlier in the data build process.

Instead, it turns out that we can actually use PowerShell instead to execute the command when the length exceeds CMD's limit. (PowerShell is available on Windows 7 and above, which is the minimum version of Windows that ICU currently supports).

Note that we only temporarily switch over to use PowerShell for the command if it exceeds the CMD limit, and then switch back to CMD for other commands. From testing, I found that using PowerShell was actually much slower, so we'd only want to use it when needed.

Checklist
…when command length exceeds CMD's limit.
@jefgen jefgen merged commit c28505c into unicode-org:master Aug 29, 2019
16 checks passed
16 checks passed
Azure CI Builds Build #20190827.14 succeeded
Details
Azure CI Builds (C: Linux Clang (Ubuntu 16.04)) C: Linux Clang (Ubuntu 16.04) succeeded
Details
Azure CI Builds (C: Linux Clang DataFilter (Ubuntu 16.04)) C: Linux Clang DataFilter (Ubuntu 16.04) succeeded
Details
Azure CI Builds (C: Linux Clang WarningsAsErrors (Ubuntu 16.04)) C: Linux Clang WarningsAsErrors (Ubuntu 16.04) succeeded
Details
Azure CI Builds (C: MSVC 32-bit Debug (VS 2017)) C: MSVC 32-bit Debug (VS 2017) succeeded
Details
Azure CI Builds (C: MSVC 64-bit Release (VS 2015)) C: MSVC 64-bit Release (VS 2015) succeeded
Details
Azure CI Builds (C: MSVC 64-bit Release (VS 2017)) C: MSVC 64-bit Release (VS 2017) succeeded
Details
Azure CI Builds (C: MSVC 64-bit Release DataFilter (VS 2017)) C: MSVC 64-bit Release DataFilter (VS 2017) succeeded
Details
Azure CI Builds (C: MSVC x64 ARM32 ARM64 Release (VS 2017)) C: MSVC x64 ARM32 ARM64 Release (VS 2017) succeeded
Details
Azure CI Builds (C: MSYS2 GCC x86_64 Release) C: MSYS2 GCC x86_64 Release succeeded
Details
Azure CI Builds (J: Linux OpenJDK (Ubuntu 16.04)) J: Linux OpenJDK (Ubuntu 16.04) succeeded
Details
continuous-integration/appveyor/pr AppVeyor build succeeded
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
jira-ticket ICU-20555 “ICU4C build fails on Windows when building from a long folder path”
Details
license/cla Contributor License Agreement is signed.
Details
single-commit This PR includes exactly 1 commit!
@jefgen jefgen deleted the jefgen:jefgen/ICU-20555-powershell-workaround branch Aug 29, 2019
@sffc

This comment has been minimized.

Copy link
Member

sffc commented Sep 4, 2019

LGTM, thanks for finding a solution to this

@linrongbin16

This comment has been minimized.

Copy link

linrongbin16 commented Dec 6, 2019

Hi, which released version will have this patch?

@jefgen

This comment has been minimized.

Copy link
Member Author

jefgen commented Dec 6, 2019

Hello @linrongbin16.
ICU 65.1 and above will have this patch. (http://site.icu-project.org/download/65).

@linrongbin16

This comment has been minimized.

Copy link

linrongbin16 commented Dec 7, 2019

Thanks!
I'm trying building ICU4C on windows10 with visual studio 2017, but failed. My environment are:

  • Windows 10 version 1909 (virtual machine running on MacOSX with vmware fusion)
  • Visual Studio 2017 Community
  • (NOTICE: I haven't install python2 or python3)

My building steps are:

  • download 65.1 source code icu4c-65_1-src.zip from icu github release page
  • extract zip and open msvc project icu4c-65_1-src\icu\source\allinone\allinone.sln
  • remove sub project makedata_uwp (no need), common_uwp, i18n_uwp (load error).
  • choose build option: Debug and x64.
  • build

I got an error message from sub project makedata:

29> set PYTHONPATH=C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\python;%PYTHONPATH
29> py -3 -B -m icutools.databuilder  --mode windows-exec  --src_dir "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data"  --tool_dir "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\tools"  --tool_cfg "x64\Debug"  --out_dir "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\out\build\icudt65l"  --tmp_dir "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\out\tmp"  --filter_file ""
29>'py' 不是内部或外部命令,也不是可运行的程序
29>或批处理文件。
29>NMAKE : fatal error U1077: “py”: 返回代码“0x1”
29>Stop.
29>C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.MakeFile.Targets(44,5): error MSB3073: 命令“NMAKE /f makedata.mak ICUMAKE="C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\\" CFG=x64\Debug”已退出,代码为 2。
29>已完成生成项目“makedata.vcxproj”的操作 - 失败。

(The error message is chinese) Translated to English, it means (translated) :

29> set PYTHONPATH=C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\python;%PYTHONPATH
29> py -3 -B -m icutools.databuilder  --mode windows-exec  --src_dir "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data"  --tool_dir "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\tools"  --tool_cfg "x64\Debug"  --out_dir "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\out\build\icudt65l"  --tmp_dir "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\out\tmp"  --filter_file ""
29>'py'  is not an inner or outer command, not an executable or batch file.
29>NMAKE : fatal error U1077: “py”:  return code “0x1”
29>Stop.
29>C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.MakeFile.Targets(44,5): error MSB3073: command “NMAKE /f makedata.mak ICUMAKE="C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\\" CFG=x64\Debug” exited, return code 2.
29>finished building project“makedata.vcxproj” - failed.

This is almost the same with this link: https://unicode-org.atlassian.net/browse/ICU-20555.

Is there anything wrong with my building steps?
Thanks.

@jefgen

This comment has been minimized.

Copy link
Member Author

jefgen commented Dec 7, 2019

Do you have Python 3 installed on your PC?

This error would seem to indicate that Python 3(or the py program) isn't installed:
29>'py' is not an inner or outer command, not an executable or batch file.

@linrongbin16

This comment has been minimized.

Copy link

linrongbin16 commented Dec 7, 2019

Thanks, I've installed python-3.7.5-amd64.exe with for all users and add python.exe to PATH options.

Then build again, I've got a new error:

1>ICU data make path is C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\
1>ICU root path is C:\Users\rolin\workspace\icu4c-65_1-src\icu
1>ICU data DLL_OUTPUT path is C:\Users\rolin\workspace\icu4c-65_1-src\icu\bin64
1>ICU tools path is C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\tools
1>ICU tools CFG subpath is x64\Debug
1>ICU data source archive is C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\in\icudt65l.dat
1>	set PYTHONPATH=C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\python;%PYTHONPATH
1>	py -3 -B -m icutools.databuilder  --mode windows-exec  --src_dir "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data"  --tool_dir "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\tools"  --tool_cfg "x64\Debug"  --out_dir "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\out\build\icudt65l"  --tmp_dir "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\out\tmp"  --filter_file ""
1>Traceback (most recent call last):
1>  File "C:\Program Files\Python37\lib\runpy.py", line 193, in _run_module_as_main
1>    "__main__", mod_spec)
1>  File "C:\Program Files\Python37\lib\runpy.py", line 85, in _run_code
1>    exec(code, run_globals)
1>  File "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\python\icutools\databuilder\__main__.py", line 347, in <module>
1>    exit(main(sys.argv[1:]))
1>  File "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\python\icutools\databuilder\__main__.py", line 291, in main
1>    requests = BUILDRULES.generate(config, glob, common)
1>  File "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\BUILDRULES.py", line 53, in generate
1>    [])
1>  File "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\BUILDRULES.py", line 461, in generate_tree
1>    input_files.remove(InFile("curr/supplementalData.txt"))
1>ValueError: list.remove(x): x not in list
1>NMAKE : fatal error U1077: “C:\Windows\py.EXE”: 返回代码“0x1”
1>Stop.
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.MakeFile.Targets(44,5): error MSB3073: 命令“NMAKE /f makedata.mak ICUMAKE="C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\\" CFG=x64\Debug”已退出,代码为 2。
1>已完成生成项目“makedata.vcxproj”的操作 - 失败。

Translated to English:

1>ICU data make path is C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\
1>ICU root path is C:\Users\rolin\workspace\icu4c-65_1-src\icu
1>ICU data DLL_OUTPUT path is C:\Users\rolin\workspace\icu4c-65_1-src\icu\bin64
1>ICU tools path is C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\tools
1>ICU tools CFG subpath is x64\Debug
1>ICU data source archive is C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\in\icudt65l.dat
1>	set PYTHONPATH=C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\python;%PYTHONPATH
1>	py -3 -B -m icutools.databuilder  --mode windows-exec  --src_dir "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data"  --tool_dir "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\tools"  --tool_cfg "x64\Debug"  --out_dir "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\out\build\icudt65l"  --tmp_dir "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\out\tmp"  --filter_file ""
1>Traceback (most recent call last):
1>  File "C:\Program Files\Python37\lib\runpy.py", line 193, in _run_module_as_main
1>    "__main__", mod_spec)
1>  File "C:\Program Files\Python37\lib\runpy.py", line 85, in _run_code
1>    exec(code, run_globals)
1>  File "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\python\icutools\databuilder\__main__.py", line 347, in <module>
1>    exit(main(sys.argv[1:]))
1>  File "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\python\icutools\databuilder\__main__.py", line 291, in main
1>    requests = BUILDRULES.generate(config, glob, common)
1>  File "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\BUILDRULES.py", line 53, in generate
1>    [])
1>  File "C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\BUILDRULES.py", line 461, in generate_tree
1>    input_files.remove(InFile("curr/supplementalData.txt"))
1>ValueError: list.remove(x): x not in list
1>NMAKE : fatal error U1077: “C:\Windows\py.EXE”: return code “0x1”
1>Stop.
1>C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.MakeFile.Targets(44,5): error MSB3073: command “NMAKE /f makedata.mak ICUMAKE="C:\Users\rolin\workspace\icu4c-65_1-src\icu\source\data\\" CFG=x64\Debug” already exited, return code 2.
1>finished building sub project “makedata.vcxproj” - failed.

Thanks

@jefgen

This comment has been minimized.

Copy link
Member Author

jefgen commented Dec 7, 2019

Hm, this looks like a bug, as the download should work...
As a work-around, are you able to use git to clone the branch for ICU version 65?

You can use the following command to clone just the maint/maint-65 branch for 65:
git clone --single-branch --branch maint/maint-65 git@github.com:unicode-org/icu.git

@linrongbin16

This comment has been minimized.

Copy link

linrongbin16 commented Dec 7, 2019

I've tried branch maint/maint-65 and build successfully!

And I've also tried tag release-65-1 with option Debug x64, which has error message:

29>	set PYTHONPATH=C:\Users\rolin\workspace\github\icu\icu4c\source\python;%PYTHONPATH
29>	py -3 -B -m icutools.databuilder  --mode windows-exec  --src_dir "C:\Users\rolin\workspace\github\icu\icu4c\source\data"  --tool_dir "C:\Users\rolin\workspace\github\icu\icu4c\source\tools"  --tool_cfg "x64\Debug"  --out_dir "C:\Users\rolin\workspace\github\icu\icu4c\source\data\out\build\icudt65l"  --tmp_dir "C:\Users\rolin\workspace\github\icu\icu4c\source\data\out\tmp"  --filter_file ""
29>!!! ERROR executing above command line: exit code 1
29>NMAKE : fatal error U1077: “C:\Windows\py.EXE”: 返回代码“0x1”
29>Stop.
29>C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.MakeFile.Targets(44,5): error MSB3073: 命令“NMAKE /f makedata.mak ICUMAKE="C:\Users\rolin\workspace\github\icu\icu4c\source\data\\" CFG=x64\Debug”已退出,代码为 2。
29>已完成生成项目“makedata.vcxproj”的操作 - 失败。
30>------ 已启动生成: 项目: makedata_uwp, 配置: Debug x64 ------
30>
30>Microsoft (R) 程序维护实用工具 14.16.27034.0 版
30>版权所有 (C) Microsoft Corporation。  保留所有权利。
30>
30>ICU data make path is C:\Users\rolin\workspace\github\icu\icu4c\source\data\
30>ICU root path is C:\Users\rolin\workspace\github\icu\icu4c
30>ICU data DLL_OUTPUT path is C:\Users\rolin\workspace\github\icu\icu4c\bin64UWP
30>ICU tools path is C:\Users\rolin\workspace\github\icu\icu4c\source\tools
30>ICU tools CFG subpath is x64\Debug
30>	set PYTHONPATH=C:\Users\rolin\workspace\github\icu\icu4c\source\python;%PYTHONPATH
30>	py -3 -B -m icutools.databuilder  --mode windows-exec  --src_dir "C:\Users\rolin\workspace\github\icu\icu4c\source\data"  --tool_dir "C:\Users\rolin\workspace\github\icu\icu4c\source\tools"  --tool_cfg "x64\Debug"  --out_dir "C:\Users\rolin\workspace\github\icu\icu4c\source\data\out\build\icudt65l"  --tmp_dir "C:\Users\rolin\workspace\github\icu\icu4c\source\data\out\tmp"  --filter_file ""
30>!!! ERROR executing above command line: exit code 1
30>NMAKE : fatal error U1077: “C:\Windows\py.EXE”: 返回代码“0x1”
30>Stop.
30>C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.MakeFile.Targets(44,5): error MSB3073: 命令“NMAKE /f makedata.mak ICUMAKE="C:\Users\rolin\workspace\github\icu\icu4c\source\data\\" CFG=x64\Debug UWP=UWP”已退出,代码为 2。
30>已完成生成项目“makedata_uwp.vcxproj”的操作 - 失败。
========== 生成: 成功 28 个,失败 2 个,最新 0 个,跳过 0 个 ==========

Translated:

29>	set PYTHONPATH=C:\Users\rolin\workspace\github\icu\icu4c\source\python;%PYTHONPATH
29>	py -3 -B -m icutools.databuilder  --mode windows-exec  --src_dir "C:\Users\rolin\workspace\github\icu\icu4c\source\data"  --tool_dir "C:\Users\rolin\workspace\github\icu\icu4c\source\tools"  --tool_cfg "x64\Debug"  --out_dir "C:\Users\rolin\workspace\github\icu\icu4c\source\data\out\build\icudt65l"  --tmp_dir "C:\Users\rolin\workspace\github\icu\icu4c\source\data\out\tmp"  --filter_file ""
29>!!! ERROR executing above command line: exit code 1
29>NMAKE : fatal error U1077: “C:\Windows\py.EXE”: return code “0x1”
29>Stop.
29>C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.MakeFile.Targets(44,5): error MSB3073: command “NMAKE /f makedata.mak ICUMAKE="C:\Users\rolin\workspace\github\icu\icu4c\source\data\\" CFG=x64\Debug” already exit, return code is 2.
29>finished building sub project “makedata.vcxproj” operation - failed.
30>------ start building: project: makedata_uwp, config: Debug x64 ------
30>
30>Microsoft (R) 程序维护实用工具 14.16.27034.0 版
30>版权所有 (C) Microsoft Corporation。  保留所有权利。
30>
30>ICU data make path is C:\Users\rolin\workspace\github\icu\icu4c\source\data\
30>ICU root path is C:\Users\rolin\workspace\github\icu\icu4c
30>ICU data DLL_OUTPUT path is C:\Users\rolin\workspace\github\icu\icu4c\bin64UWP
30>ICU tools path is C:\Users\rolin\workspace\github\icu\icu4c\source\tools
30>ICU tools CFG subpath is x64\Debug
30>	set PYTHONPATH=C:\Users\rolin\workspace\github\icu\icu4c\source\python;%PYTHONPATH
30>	py -3 -B -m icutools.databuilder  --mode windows-exec  --src_dir "C:\Users\rolin\workspace\github\icu\icu4c\source\data"  --tool_dir "C:\Users\rolin\workspace\github\icu\icu4c\source\tools"  --tool_cfg "x64\Debug"  --out_dir "C:\Users\rolin\workspace\github\icu\icu4c\source\data\out\build\icudt65l"  --tmp_dir "C:\Users\rolin\workspace\github\icu\icu4c\source\data\out\tmp"  --filter_file ""
30>!!! ERROR executing above command line: exit code 1
30>NMAKE : fatal error U1077: “C:\Windows\py.EXE”: return code “0x1”
30>Stop.
30>C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE\VC\VCTargets\Microsoft.MakeFile.Targets(44,5): error MSB3073: 命令“NMAKE /f makedata.mak ICUMAKE="C:\Users\rolin\workspace\github\icu\icu4c\source\data\\" CFG=x64\Debug UWP=UWP” already exit, return code is 2.
30>finished building project “makedata_uwp.vcxproj” operation - failed.
========== building: success 28, failed 2, newest 0, skip 0 ==========

Thanks!

@linrongbin16

This comment has been minimized.

Copy link

linrongbin16 commented Dec 8, 2019

Hi, I've tried to git clone release-64-2 at device root C:\icu. then build successfully.
Thanks.

@jefgen

This comment has been minimized.

Copy link
Member Author

jefgen commented Dec 8, 2019

I've tried branch maint/maint-65 and build successfully!
And I've also tried tag release-65-1 with option Debug x64, which has error message:

If you're getting an error with the tag but not the branch then I'd suspect that maybe the clone or checkout didn't work properly.

Note that both maint/maint-65 and release-65-1 point to the exact same commit, so there should be absolutely no difference between them.

You can check on the command line, or on GitHub by looking at the commits here:
https://github.com/unicode-org/icu/commits/release-65-1
https://github.com/unicode-org/icu/commits/maint/maint-65

That said, generally speaking, I would really suggest using using the "maint" branches instead of the release tags. The "maint" branches have the same version as a given release, but since they are "maintenance" branches they will often contain additional commits for fixes for bugs and/or security issues that are discovered after a release is published. (Though in this case there is currently nothing additional on the maint branch for ICU 65).

For example, for ICU 64, the maint/maint-64 branch currently contains 3 extra commits on top of the release-64-2 tag: release-64-2...maint/maint-64

Using a shorter folder path (like checking out to device root) is a work-around for this bug, so you could use ICU 64 with this workaround if that works for you. :)

@linrongbin16

This comment has been minimized.

Copy link

linrongbin16 commented Dec 8, 2019

Thank you!

@zavexu

This comment has been minimized.

Copy link

zavexu commented Dec 18, 2019

Using a shorter folder path (like checking out to device root) is a work-around for this bug, so you could use ICU 64 with this workaround if that works for you. :)

Yes :), I move the icu4c folder to device root C:\, that's C:\icu4c. It works as expect for ICU 65.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
6 participants
You can’t perform that action at this time.