Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Release buildtools for Windows as well #375

Closed
oferb opened this issue Sep 16, 2018 · 22 comments
Closed

Release buildtools for Windows as well #375

oferb opened this issue Sep 16, 2018 · 22 comments

Comments

@oferb
Copy link

oferb commented Sep 16, 2018

The repo releases:
https://github.com/bazelbuild/buildtools/releases

only contain Linux and Mac binaries. Would be nice to also have Windows there :-)
@oferb oferb mentioned this issue Sep 16, 2018
Closed
@laszlocsomor
Copy link
Contributor

+1

I successfully built Buildozer and Buildifier at 220349f using Bazel 0.18.0 on Windows.

@pmbethe09
Copy link
Member

I would be very happy to include Windows builds as part of an automated release process.
However we don't have one yet -- no one has had the time to follow up on this.
Right now I do linux/Mac builds manually and don't have access to a Windows platform to do that build.

@oferb
Copy link
Author

oferb commented Oct 20, 2018

What constitutes a release process? Just building it, or are there additional steps?

@oferb
Copy link
Author

oferb commented Oct 20, 2018

BTW - for my particular use, I'm going for WSL - Windows Subsystem for Linux, so currently I don't support plain Windows for https://github.com/google/startup-os.

@laurentlb
Copy link
Contributor

cc @philwo for any recommendation (is it easy to automate? could it be part of the Bazel release process?)
cc @aiuto

@philwo
Copy link
Member

philwo commented Oct 22, 2018

This is difficulty is that we only have two sets of machines where we can build releases on:

  • The "trusted" Buildkite organization that will build Bazel releases in the future.
  • The "untrusted" Buildkite organization where all other CI jobs run.

Only code that has been 100% reviewed by Bazel core team members can run in the trusted zone in order to reduce risk of infecting the machines with malware and then accidentally shipping an infected version of Bazel to our users.

You could run release builds in the untrusted zone and while I think we've done a pretty good job of keeping these machines clean, stateless and somewhat trustable, in theory someone could escape the sandbox on these machines and affect builds that come afterwards.

On Linux / Windows we could somehow solve this by never recycling machines (always delete and start a fresh one), but for macOS we only have physical machines that we can't easily wipe clean after each job.

Maybe that's still better than any other option. We can look into it, but currently have to do other tasks first.

@evankanderson
Copy link

It sounds like we already have release builds for macOS (which mentions physical machines as a limit), but not Windows. I'm guessing the problem with Windows is getting the automation set up?

I'm running into this trying to build https://github.com/jetstack/cert-manager, which depends on these release binaries through some platform-matching magic which fails on Windows. I'm building on a VM in the meantime, but it would be nice to have a faster edit-compile cycle that didn't involve syncing files between machines. 😬

@aiuto
Copy link
Contributor

aiuto commented Nov 26, 2018 via email

@alexeagle
Copy link
Contributor

Ping, we are using buildifier on Angular team now and our Windows devs are getting tripped up on formatting their changes to satisfy the CI

/cc @ocombe

@aiuto
Copy link
Contributor

aiuto commented Dec 20, 2018 via email

@alexeagle
Copy link
Contributor

For Angular's developers, I'd be willing to manually create windows releases to publish to npm, at least for a short time.
However, I get an error trying to build at the 0.20.0 tag on Windows:

Internal error thrown during build. Printing stack trace: java.lang.RuntimeException: Unrecoverable error while evaluating node 'REPOSITORY_DIRECTORY:@org_golang_x_tools' (requested by nodes 'REPOSITORY:@org_golang_x_tools')
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:495)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:368)
        at java.base/java.util.concurrent.ForkJoinTask$AdaptedRunnableAction.exec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.localPopAndExec(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)

@laszlocsomor
Copy link
Contributor

@alexeagle : which Bazel version do you use and what command did you run?

With Bazel 0.21.0 and at the 0.20.0 release tag (db07345), I can build Buildifier2:

First, in Git Bash:

laszlocsomor@laszlocsomor2-w MINGW64 /c/src/buildtools ((0.20.0))
$ git rev-parse HEAD
db073457c5a56d810e46efc18bb93a4fd7aa7b5e

Then, in cmd.exe:

C:\src\buildtools>bazel build //buildifier2
WARNING: The following rc files are no longer being read, please transfer their contents or import their path into one of the standard rc files:
C:\Users\laszlocsomor/.bazelrc
...
Target //buildifier2:buildifier2 up-to-date:
  C:/_bazel/ctlg6o7x/execroot/com_github_bazelbuild_buildtools/bazel-out/x64_windows-fastbuild/bin/buildifier2/windows_amd64_stripped/buildifier2.exe
INFO: Elapsed time: 37.536s, Critical Path: 4.61s
INFO: 13 processes: 13 local.
INFO: Build completed successfully, 19 total actions

(For the record, I had to install diffutils and git under MSYS2, by running pacman -S diffutils git, and set my PATH in cmd.exe to set path=c:\msys64\usr\bin;c:\src\bazel-releases\current;c:\Python37;c:\Windows\System32. This forced Bazel to use MSYS2 Bash with MSYS2's Git, instead of using Git Bash with the Git-for-Windows which lacks patch and also lacks Pacman so you cannot even install packages.)

@meteorcloudy
Copy link
Member

meteorcloudy commented Feb 25, 2019 •
edited

Buildtools project has been running and passing on Windows since #522 was merged.
https://buildkite.com/bazel/buildtools/builds/826#90d7a980-81b6-45f0-ae90-52745648cf79

@vladmos How does buildtools got released, can we add Windows to the release now?

@philwo
Copy link
Member

philwo commented Feb 25, 2019

@aiuto I don't know why I completely missed the ongoing discussion here on this bug. Sorry :(

Yes, of course, let's get something working so we can produce code-signed Windows binaries for Bazel. We can securely store and use signing keys in our cloud project and use them from the trusted Windows machines.

I'm not familiar with the tooling needed to do the signing on Windows, but it sounds like you are.

I set the e-mail that just notified me about this issue to pop-up in my inbox when I'm back from vacation on April 2nd. Let's discuss then?

@aiuto
Copy link
Contributor

aiuto commented Feb 25, 2019 via email

@vladmos vladmos mentioned this issue Mar 25, 2019
Closed
@aiuto
Copy link
Contributor

aiuto commented Apr 1, 2019

Time to pick this (and general signing issues) up again.

@aiuto aiuto closed this as completed Apr 1, 2019
@aiuto aiuto reopened this Apr 1, 2019
@aiuto
Copy link
Contributor

aiuto commented Apr 1, 2019

I have some things I can point you to about signing windows builds and verifying that they were signed correctly. But they are internal code so I can't put the link here. It's easy for me to point you at the relevant things so you could do an equivalent. At your convenience.

@philwo philwo mentioned this issue Apr 4, 2019
Closed
@philwo
Copy link
Member

philwo commented Apr 4, 2019

@aiuto Unfortunately I'm on sick-leave this week, but let's get this done soon. I've added a tracking bug in our CI repo so that I don't forget about this.

Can you send me pointers to how this would work? I guess the rough flow is:

  • We have to get a code-signing certificate from somewhere.
  • We have to store this encrypted securely in our trusted GCP project.
  • We create a pipeline to do buildtools releases via Buildkite.
  • We add the step that uses the code-signing certificate to turn the unsigned .exe into a signed .exe and then release that one. (I guess signing a file is a matter of calling some command-line tool?)

Is that about right?

@aiuto
Copy link
Contributor

aiuto commented Apr 4, 2019 via email

@vladmos vladmos mentioned this issue May 15, 2019
Merged
@vladmos
Copy link
Member

vladmos commented May 15, 2019

Windows binaries are available starting with 0.25.1 released today: https://github.com/bazelbuild/buildtools/releases/tag/0.25.1

They are supposed to be fully functional, however it the first time we've released Windows binaries, please let us know if there are any platform-specific issues.

@vladmos vladmos closed this as completed May 15, 2019
@saxicek
Copy link

saxicek commented Jan 20, 2021

@vladmos I think it would be good to update the following docs - https://docs.bazel.build/versions/master/build-javascript.html#step-4-linting
At the end it claims:

Also the Buildifier tool is not available on Windows. Follow #375

bazel-io pushed a commit to bazelbuild/bazel that referenced this issue Jan 20, 2021
@meteorcloudy
Fix doc about Buildifier Windows support
ec2aa5e 
Related bazelbuild/buildtools#375

RELNOTES: None
PiperOrigin-RevId: 352787916
@vladmos
Copy link
Member

vladmos commented Jan 22, 2021

Thanks for noticing, the documentation has already been fixed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests