New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windows: Error: corrupt installation #4378

Closed
meteorcloudy opened this Issue Jan 2, 2018 · 13 comments

Comments

Projects
None yet
7 participants
@meteorcloudy
Copy link
Member

meteorcloudy commented Jan 2, 2018

I noticed this error in many places, my local machine, Bazel CI, Tensorflow CI,

Error: corrupt installation: file 'C:\tmp/_bazel_pcloudy/install/6f94f50e4679b4d40059370500e5cb9c/_embedded_binaries/A-server.jar' modified.  Please remove 'C:\tmp/_bazel_pcloudy/install/6f94f50e4679b4d40059370500e5cb9c' and try again.

The interesting part is that, it happens with every old Bazel versions in the new year!

The file timestamps are all set to Jan 1 2027.

pcloudy@tensorflow-jenkins-win-gpu2-slave MSYS ~/workspace/tensorflow
$ ll /c/tmp/_bazel_pcloudy/install/6f94f50e4679b4d40059370500e5cb9c/_embedded_binaries
total 44894
-rw-r--r-- 1 pcloudy None 45074399 Jan  1  2027 A-server.jar
-rwxr-xr-x 1 pcloudy None   257536 Jan  1  2027 build-runfiles.exe
drwxr-xr-x 1 pcloudy None        0 Oct 19 16:13 embedded_tools
-rw-r--r-- 1 pcloudy None       33 Jan  1  2027 install_base_key
-rw-r--r-- 1 pcloudy None        4 Jan  1  2027 java.version
-rw-r--r-- 1 pcloudy None     3849 Jan  1  2027 jdk.BUILD
-rwxr-xr-x 1 pcloudy None    90112 Jan  1  2027 linux-sandbox.exe
-rwxr-xr-x 1 pcloudy None   257536 Jan  1  2027 process-wrapper.exe
-rwxr-xr-x 1 pcloudy None   272896 Jan  1  2027 windows_jni.dll
-rwxr-xr-x 1 pcloudy None      698 Jan  1  2027 xcode-locator

@laszlocsomor Do you have any idea about this? Is there anything wrong with our file invalid mechanism?
FYI @mhlopko

@meteorcloudy meteorcloudy changed the title Error: corrupt installation: file '.../install/6f94f50e4679b4d40059370500e5cb9c/_embedded_binaries/A-server.jar' modified Windows: Error: corrupt installation: file '.../install/6f94f50e4679b4d40059370500e5cb9c/_embedded_binaries/A-server.jar' modified Jan 2, 2018

@meteorcloudy meteorcloudy changed the title Windows: Error: corrupt installation: file '.../install/6f94f50e4679b4d40059370500e5cb9c/_embedded_binaries/A-server.jar' modified Windows: Error: corrupt installation Jan 2, 2018

@wyan-nestlabs

This comment has been minimized.

Copy link

wyan-nestlabs commented Jan 2, 2018

We ran into the same problem in all our own windows builder machines. Removing the following two directories fixed the problem: C:\Users\builder\_bazel_LOCAL\ SERVICE/install/dc4fe1b7fd90df4d3976b7aaa4b41899 and cygwin /tmp/_bazel_LOCAL\ SERVICE/install/dc4fe1b7fd90df4d3976b7aaa4b41899. But it is not addressing the reason for the failure.

PS, the same failure came back. I had to delete the two directories again. The hash in the paths stayed the same.

@laszlocsomor

This comment has been minimized.

Copy link
Contributor

laszlocsomor commented Jan 3, 2018

@meteorcloudy : This is Windows only, right? Then it's likely a bug in this class: https://github.com/bazelbuild/bazel/blob/master/src/main/cpp/util/file_windows.cc#L156 . Yesterday was Jan 2, did you see this yesterday too, or only on Jan 1, or with Bazel installations from last year?

@meteorcloudy

This comment has been minimized.

Copy link
Member

meteorcloudy commented Jan 3, 2018

I saw it on Jan 2 and also today Jan 3, and yes, with every bazel installations from last year.

@laszlocsomor

This comment has been minimized.

Copy link
Contributor

laszlocsomor commented Jan 3, 2018

Yeah the code looks really suspicious. Let me take a closer look.

@meteorcloudy

This comment has been minimized.

Copy link
Member

meteorcloudy commented Jan 3, 2018

@mhlopko To fix the Windows CI build, we'll have to delete the installation directories.

@meteorcloudy

This comment has been minimized.

Copy link
Member

meteorcloudy commented Jan 3, 2018

@laszlocsomor Thanks!

@laszlocsomor

This comment has been minimized.

Copy link
Contributor

laszlocsomor commented Jan 3, 2018

One workaround is to delete the install directory (dirname of "A-server.jar" in the error message).
Another workaround is to run touch -m -t 202801010101 $(find path/to/install/dir -type f) in MSYS shell.

@laszlocsomor

This comment has been minimized.

Copy link
Contributor

laszlocsomor commented Jan 3, 2018

Another workaround: rename the install directory.
This is the easiest approach. It's useful if deleting fails due to long file names. (Delete those directories from MSYS or using Total Commander.)

@rongjiecomputer

This comment has been minimized.

Copy link
Contributor

rongjiecomputer commented Jan 4, 2018

I hit this bug in my local Windows 10 machine as well on Jan 3. Some Bazel files have the "01/01/2028" timestamp.

Reinstalling Bazel fixes the issue, but new Bazel files still have the timestamp of 01/01/2028. See screenshots.

capture

@laszlocsomor

This comment has been minimized.

Copy link
Contributor

laszlocsomor commented Jan 4, 2018

Reinstalling Bazel fixes the issue, but new Bazel files still have the timestamp of 01/01/2028. See screenshots.

That is working as intended.

When you install Bazel, Bazel unpacks embedded binaries and sets their timestamp to Jan 1, 10+$current_year. It's 2018 now, so the timestamps are 2028/01/01.

@buchgr

This comment has been minimized.

Copy link
Contributor

buchgr commented Mar 21, 2018

Can this be closed? Please re-open if still an issue.

@laszlocsomor

This comment has been minimized.

Copy link
Contributor

laszlocsomor commented Mar 21, 2018

No, this bug is not yet fixed.

@laszlocsomor laszlocsomor reopened this Mar 21, 2018

@flaub

This comment has been minimized.

Copy link

flaub commented Mar 29, 2018

Just ran into this issue here.

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Jun 13, 2018

Windows: fix "corrupt installation" at new year
Fixes bazelbuild#4378

Change-Id: I3457bdc360a62a279d1d08c9a69997929f2067dd

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Jun 13, 2018

Windows: fix "corrupt installation" at new year
Fixes bazelbuild#4378

Change-Id: I3457bdc360a62a279d1d08c9a69997929f2067dd

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Jun 13, 2018

Windows: fix "corrupt installation" at new year
Bazel on Windows is now consistent with Bazel on
Unixes, by setting the mtimes of embedded binaries
to 10 years in the future.

Before this change, on Windows, Bazel used to set
these mtimes to CURRENT_YEAR + 10, January 1st.

This meant that if a user ran Bazel on 2017/12/29,
then on Unix Bazel set the mtimes to 2027/12/29
but on Windows to 2027/01/01.

If the user then ran Bazel in the same workspace
on 2018/01/02, on Unixes it worked fine, but on
Windows it detected that the embedded binaries'
mtime is older than 2018/01/01, and reported a
"corrupt installation" error.

Fixes bazelbuild#4378

Change-Id: I3457bdc360a62a279d1d08c9a69997929f2067dd

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Jun 13, 2018

Windows: fix "corrupt installation" at new year
Bazel on Windows is now consistent with Bazel on
Unixes, by setting the mtimes of embedded binaries
to 10 years in the future, instead of setting them
to January 1st in the 10th year from now.

On Unix, Bazel sets the mtime of embedded binaries
to 10 years into the future. At startup, it checks
that their mtime is at least 9 years in the
future, for two reasons:

- to detect unintentional tampering with the
  embedded binaries, which would reset their mtime
  to the current time, and

- and because we don't expect users to use the
  same version of Bazel in the same workspace for
  more than a year.

On Windows, Bazel had a bug: instead of setting
the mtime to the current time + 10 years, and
comparing them to current time + 9 years, it was
setting current year + 10 years, January 1st, and
comparing to current year + 9 years, January 1st.

This meant that if a user ran Bazel for the first
time on 2017/12/29, then ran again on 2018/01/02,
then:

- on Unixes, the mtimes were set to 2027/12/29,
  then later compared to 2027/01/02, but

- on Windows, the mtimes were set to 2027/01/01,
  then later compared to 2027/01/01, of which
  they were not newer.

Fixes bazelbuild#4378

Change-Id: I3457bdc360a62a279d1d08c9a69997929f2067dd

laszlocsomor added a commit to laszlocsomor/bazel that referenced this issue Jun 13, 2018

Windows: fix "corrupt installation" at new year
Bazel on Windows is now consistent with Bazel on
Unixes, by setting the mtimes of embedded binaries
to 10 years in the future, instead of setting them
to January 1st in the 10th year from now.

On Unix, Bazel sets the mtime of embedded binaries
to 10 years into the future. At startup, it checks
that their mtime is at least 9 years in the
future, for two reasons:

- to detect unintentional tampering with the
  embedded binaries, which would reset their mtime
  to the current time, and

- and because we don't expect users to use the
  same version of Bazel in the same workspace for
  more than a year.

On Windows, Bazel had a bug: instead of setting
the mtime to the current time + 10 years, and
comparing them to current time + 9 years, it was
setting current year + 10 years, January 1st, and
comparing to current year + 9 years, January 1st.

This meant that if a user ran Bazel for the first
time on 2017/12/29, then ran again on 2018/01/02,
then:

- on Unixes, the mtimes were set to 2027/12/29,
  then later compared to 2027/01/02, but

- on Windows, the mtimes were set to 2027/01/01,
  then later compared to 2027/01/01, of which
  they were not newer.

Fixes bazelbuild#4378

Change-Id: I3457bdc360a62a279d1d08c9a69997929f2067dd

@bazel-io bazel-io closed this in 6b43357 Jun 13, 2018

@laszlocsomor laszlocsomor self-assigned this Jun 14, 2018

ArielleA added a commit to ArielleA/bazel that referenced this issue Jun 19, 2018

Windows: fix "corrupt installation" at new year
Bazel on Windows is now consistent with Bazel on
Unixes, by setting the mtimes of embedded binaries
to 10 years in the future.

Before this change, on Windows, Bazel used to set
these mtimes to CURRENT_YEAR + 10, January 1st.

This meant that if a user ran Bazel on 2017/12/29,
then on Unix Bazel set the mtimes to 2027/12/29
but on Windows it set them to 2027/01/01.

If the user then ran Bazel in the same workspace
on 2018/01/02, on Unixes it worked fine, but on
Windows it detected that the embedded binaries'
mtime is older than 2018/01/01, and reported a
"corrupt installation" error.

Fixes bazelbuild#4378

Change-Id: I3457bdc360a62a279d1d08c9a69997929f2067dd

Closes bazelbuild#5385.

Change-Id: I3457bdc360a62a279d1d08c9a69997929f2067dd
PiperOrigin-RevId: 200395493

werkt added a commit to werkt/bazel that referenced this issue Aug 2, 2018

Windows: fix "corrupt installation" at new year
Bazel on Windows is now consistent with Bazel on
Unixes, by setting the mtimes of embedded binaries
to 10 years in the future.

Before this change, on Windows, Bazel used to set
these mtimes to CURRENT_YEAR + 10, January 1st.

This meant that if a user ran Bazel on 2017/12/29,
then on Unix Bazel set the mtimes to 2027/12/29
but on Windows it set them to 2027/01/01.

If the user then ran Bazel in the same workspace
on 2018/01/02, on Unixes it worked fine, but on
Windows it detected that the embedded binaries'
mtime is older than 2018/01/01, and reported a
"corrupt installation" error.

Fixes bazelbuild#4378

Change-Id: I3457bdc360a62a279d1d08c9a69997929f2067dd

Closes bazelbuild#5385.

Change-Id: I3457bdc360a62a279d1d08c9a69997929f2067dd
PiperOrigin-RevId: 200395493
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment