Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Help me get the homebrew formula building again #851

Open
isker opened this issue Aug 25, 2020 · 16 comments
Open

Help me get the homebrew formula building again #851

isker opened this issue Aug 25, 2020 · 16 comments

Comments

@isker
Copy link

isker commented Aug 25, 2020

Hello @wez! As per #665 (comment) I am trying to get the homebrew formula building again. Sadly my knowledge of: homebrew internals; watchman; and native toolchains of any kind are all pretty poor 🙂 .

I made the suggested changes to the formula:

modified   Formula/watchman.rb
@@ -1,21 +1,12 @@
 class Watchman < Formula
   desc "Watch files and take action when they change"
   homepage "https://github.com/facebook/watchman"
+  url "https://github.com/facebook/watchman/archive/v2020.08.17.00.tar.gz"
+  sha256 "9229687ee834b4527746f61a8071a9d1bb16415e948787120703c5f511383e6e"
   license "Apache-2.0"
-  revision 4
+  version_scheme 1
   head "https://github.com/facebook/watchman.git"
 
-  stable do
-    url "https://github.com/facebook/watchman/archive/v4.9.0.tar.gz"
-    sha256 "1f6402dc70b1d056fffc3748f2fdcecff730d8843bb6936de395b3443ce05322"
-
-    # Upstream commit from 1 Sep 2017: "Have bin scripts use iter() method for python3"
-    patch do
-      url "https://github.com/facebook/watchman/commit/17958f7d.diff?full_index=1"
-      sha256 "edad4971fceed2aecfa2b9c3e8e22c455bfa073732a3a0c77b030e506ee860af"
-    end
-  end
-
   bottle do
     sha256 "7840f564c11d33425c9eb8985f9156e782e66ef2a3578329dba83ee15a9bf0be" => :catalina
     sha256 "ba2338b0f23c8b8817fd7bfa92466b7a97ab416e93ec6c3a400041aef013de86" => :mojave
@@ -26,6 +17,7 @@ class Watchman < Formula
   depends_on "automake" => :build
   depends_on "libtool" => :build
   depends_on "pkg-config" => :build
+  depends_on "folly"
   depends_on "openssl@1.1"
   depends_on "pcre"
   depends_on "python@3.8"

But getdeps.py is, well, still trying to get all of the deps. Right now it's trying and failing to build openssl. I'm not sure why, but I'm pretty sure it shouldn't be trying to build it in the first place.

OpenSSL build error

Assessing openssl...
Download https://www.openssl.org/source/openssl-1.1.1f.tar.gz -> /private/tmp/fbcode_builder_getdeps-ZprivateZtmpZwatchman-20200825-72628-nwvu33Zwatchman-2020.08.17.00ZbuildZfbcode_builder/downloads/openssl-openssl-1.1.1f.tar.gz ...
.. 0 of 9792828  [Complete in 0.834120 seconds]
---
+ cd /private/tmp/fbcode_builder_getdeps-ZprivateZtmpZwatchman-20200825-72628-nwvu33Zwatchman-2020.08.17.00ZbuildZfbcode_builder/build/openssl-UtXLnJ2t0v50wqKwl2XtYbrqHH1o2BVeZQYDPt0Ta_c && \
+ /usr/bin/perl \
+      /private/tmp/fbcode_builder_getdeps-ZprivateZtmpZwatchman-20200825-72628-nwvu33Zwatchman-2020.08.17.00ZbuildZfbcode_builder/extracted/openssl-openssl-1.1.1f.tar.gz/openssl-1.1.1f/Configure \
+      --prefix=/private/tmp/fbcode_builder_getdeps-ZprivateZtmpZwatchman-20200825-72628-nwvu33Zwatchman-2020.08.17.00ZbuildZfbcode_builder/installed/openssl-UtXLnJ2t0v50wqKwl2XtYbrqHH1o2BVeZQYDPt0Ta_c \
+      --openssldir=/private/tmp/fbcode_builder_getdeps-ZprivateZtmpZwatchman-20200825-72628-nwvu33Zwatchman-2020.08.17.00ZbuildZfbcode_builder/installed/openssl-UtXLnJ2t0v50wqKwl2XtYbrqHH1o2BVeZQYDPt0Ta_c \
+      darwin64-x86_64-cc \
+      enable-static-engine \
+      enable-capieng \
+      no-makedepend \
+      no-unit-test \
+      no-tests
Configuring OpenSSL version 1.1.1f (0x1010106fL) for darwin64-x86_64-cc
Using os-specific seed configuration
Creating configdata.pm
Creating Makefile

**********************************************************************
***                                                                ***
***   OpenSSL has been successfully configured                     ***
***                                                                ***
***   If you encounter a problem while building, please open an    ***
***   issue on GitHub <https://github.com/openssl/openssl/issues>  ***
***   and include the output from the following command:           ***
***                                                                ***
***       perl configdata.pm --dump                                ***
***                                                                ***
***   (If you are new to OpenSSL, you might want to consult the    ***
***   'Troubleshooting' section in the INSTALL file first)         ***
***                                                                ***
**********************************************************************
---
+ cd /private/tmp/fbcode_builder_getdeps-ZprivateZtmpZwatchman-20200825-72628-nwvu33Zwatchman-2020.08.17.00ZbuildZfbcode_builder/build/openssl-UtXLnJ2t0v50wqKwl2XtYbrqHH1o2BVeZQYDPt0Ta_c && \
+ make \
+      install_sw \
+      install_ssldirs
/usr/bin/perl "-I." -Mconfigdata "../../extracted/openssl-openssl-1.1.1f.tar.gz/openssl-1.1.1f/util/dofile.pl" \
	    "-oMakefile" ../../extracted/openssl-openssl-1.1.1f.tar.gz/openssl-1.1.1f/include/crypto/bn_conf.h.in > include/crypto/bn_conf.h
/usr/bin/perl "-I." -Mconfigdata "../../extracted/openssl-openssl-1.1.1f.tar.gz/openssl-1.1.1f/util/dofile.pl" \
	    "-oMakefile" ../../extracted/openssl-openssl-1.1.1f.tar.gz/openssl-1.1.1f/include/crypto/dso_conf.h.in > include/crypto/dso_conf.h
/usr/bin/perl "-I." -Mconfigdata "../../extracted/openssl-openssl-1.1.1f.tar.gz/openssl-1.1.1f/util/dofile.pl" \
	    "-oMakefile" ../../extracted/openssl-openssl-1.1.1f.tar.gz/openssl-1.1.1f/include/openssl/opensslconf.h.in > include/openssl/opensslconf.h
created directory `/private/tmp/fbcode_builder_getdeps-ZprivateZtmpZwatchman-20200825-72628-nwvu33Zwatchman-2020.08.17.00ZbuildZfbcode_builder/installed/openssl-UtXLnJ2t0v50wqKwl2XtYbrqHH1o2BVeZQYDPt0Ta_c'
created directory `/private/tmp/fbcode_builder_getdeps-ZprivateZtmpZwatchman-20200825-72628-nwvu33Zwatchman-2020.08.17.00ZbuildZfbcode_builder/installed/openssl-UtXLnJ2t0v50wqKwl2XtYbrqHH1o2BVeZQYDPt0Ta_c/certs'
created directory `/private/tmp/fbcode_builder_getdeps-ZprivateZtmpZwatchman-20200825-72628-nwvu33Zwatchman-2020.08.17.00ZbuildZfbcode_builder/installed/openssl-UtXLnJ2t0v50wqKwl2XtYbrqHH1o2BVeZQYDPt0Ta_c/private'
created directory `/private/tmp/fbcode_builder_getdeps-ZprivateZtmpZwatchman-20200825-72628-nwvu33Zwatchman-2020.08.17.00ZbuildZfbcode_builder/installed/openssl-UtXLnJ2t0v50wqKwl2XtYbrqHH1o2BVeZQYDPt0Ta_c/misc'
Command '['make', 'install_sw', 'install_ssldirs']' returned non-zero exit status 2.
!! Failed
install ./apps/CA.pl -> /private/tmp/fbcode_builder_getdeps-ZprivateZtmpZwatchman-20200825-72628-nwvu33Zwatchman-2020.08.17.00ZbuildZfbcode_builder/installed/openssl-UtXLnJ2t0v50wqKwl2XtYbrqHH1o2BVeZQYDPt0Ta_c/misc/CA.pl
cp: ./apps/CA.pl: No such file or directory
make: *** [install_ssldirs] Error 1
make: *** Waiting for unfinished jobs....

Am I supposed to skip autogen.sh and invoke getdeps.py directly with something like --allow-system-packages? If so, I think I'd need to add homebrew dependencies on more than just folly. Boost, for example. Is there a full list handy anywhere?

@isker
Copy link
Author

isker commented Aug 25, 2020

Nevermind Boost, it's a transitive dep of folly:

λ brew deps --include-build --tree watchman
watchman
├── autoconf
├── automake
│   └── autoconf
├── libtool
├── pkg-config
├── folly
│   ├── cmake
│   │   └── sphinx-doc
│   │       └── python@3.8
│   │           ├── pkg-config
│   │           ├── gdbm
│   │           ├── openssl@1.1
│   │           ├── readline
│   │           ├── sqlite
│   │           │   └── readline
│   │           └── xz
│   ├── pkg-config
│   ├── boost
│   │   └── icu4c
│   ├── double-conversion
│   │   └── cmake
│   │       └── sphinx-doc
│   │           └── python@3.8
│   │               ├── pkg-config
│   │               ├── gdbm
│   │               ├── openssl@1.1
│   │               ├── readline
│   │               ├── sqlite
│   │               │   └── readline
│   │               └── xz
│   ├── fmt
│   │   └── cmake
│   │       └── sphinx-doc
│   │           └── python@3.8
│   │               ├── pkg-config
│   │               ├── gdbm
│   │               ├── openssl@1.1
│   │               ├── readline
│   │               ├── sqlite
│   │               │   └── readline
│   │               └── xz
│   ├── gflags
│   │   └── cmake
│   │       └── sphinx-doc
│   │           └── python@3.8
│   │               ├── pkg-config
│   │               ├── gdbm
│   │               ├── openssl@1.1
│   │               ├── readline
│   │               ├── sqlite
│   │               │   └── readline
│   │               └── xz
│   ├── glog
│   │   ├── cmake
│   │   │   └── sphinx-doc
│   │   │       └── python@3.8
│   │   │           ├── pkg-config
│   │   │           ├── gdbm
│   │   │           ├── openssl@1.1
│   │   │           ├── readline
│   │   │           ├── sqlite
│   │   │           │   └── readline
│   │   │           └── xz
│   │   └── gflags
│   │       └── cmake
│   │           └── sphinx-doc
│   │               └── python@3.8
│   │                   ├── pkg-config
│   │                   ├── gdbm
│   │                   ├── openssl@1.1
│   │                   ├── readline
│   │                   ├── sqlite
│   │                   │   └── readline
│   │                   └── xz
│   ├── libevent
│   │   ├── autoconf
│   │   ├── automake
│   │   │   └── autoconf
│   │   ├── libtool
│   │   ├── pkg-config
│   │   └── openssl@1.1
│   ├── lz4
│   ├── openssl@1.1
│   ├── snappy
│   │   ├── cmake
│   │   │   └── sphinx-doc
│   │   │       └── python@3.8
│   │   │           ├── pkg-config
│   │   │           ├── gdbm
│   │   │           ├── openssl@1.1
│   │   │           ├── readline
│   │   │           ├── sqlite
│   │   │           │   └── readline
│   │   │           └── xz
│   │   └── pkg-config
│   ├── xz
│   └── zstd
│       └── cmake
│           └── sphinx-doc
│               └── python@3.8
│                   ├── pkg-config
│                   ├── gdbm
│                   ├── openssl@1.1
│                   ├── readline
│                   ├── sqlite
│                   │   └── readline
│                   └── xz
├── openssl@1.1
├── pcre
└── python@3.8
    ├── pkg-config
    ├── gdbm
    ├── openssl@1.1
    ├── readline
    ├── sqlite
    │   └── readline
    └── xz

Though judging by the list from CI, some deps like ninja are still missing from the formula.

- name: Fetch boost
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests boost
- name: Fetch openssl
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests openssl
- name: Fetch ninja
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests ninja
- name: Fetch cmake
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests cmake
- name: Fetch cpptoml
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests cpptoml
- name: Fetch googletest
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests googletest
- name: Fetch pcre
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests pcre
- name: Fetch fmt
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests fmt
- name: Fetch python-six
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests python-six
- name: Fetch zstd
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests zstd
- name: Fetch double-conversion
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests double-conversion
- name: Fetch gflags
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests gflags
- name: Fetch glog
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests glog
- name: Fetch libevent
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libevent
- name: Fetch lz4
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests lz4
- name: Fetch snappy
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests snappy
- name: Fetch folly
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests folly
- name: Fetch autoconf
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests autoconf
- name: Fetch automake
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests automake
- name: Fetch libtool
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libtool
- name: Fetch bison
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests bison
- name: Fetch libsodium
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests libsodium
- name: Fetch fizz
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests fizz
- name: Fetch flex
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests flex
- name: Fetch wangle
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests wangle
- name: Fetch fbthrift
run: python3 build/fbcode_builder/getdeps.py --allow-system-packages fetch --no-tests fbthrift
- name: Fetch fb303

@si14
Copy link

si14 commented Aug 25, 2020

This issue might be related/of interest: #839

@wez
Copy link
Contributor

wez commented Aug 25, 2020

Thanks for taking a look at this!

I think fixing up the homebrew formula in core homebrew is probably quite difficult.

The easiest way to build watchman from master today is using our getdeps machinery. That is responsible for fetching, building and composing the dependencies. This is at odds with homebrew's own dependency management, so a proper update to the homebrew formula probably should not use the getdeps stuff.

Another challenge here is that watchman depends on folly and fbthrift and all of those projects use a continuous delivery model in the internal single source repository where they all reside. When we export to github those projects are mapped to separate projects that are hard to relate to each other. We have another piece of infrastructure that will cut a weekly tag to try to form a consistent relationship, but due to the nature of continuous delivery, any of those projects might not actually be buildable with each other due to various transient bugs.

I've been copied on numerous issues in the folly repo over the past few years where folks have been working to resolve build issues that block updating the folly homebrew formula, so I sense that this might turn into an ongoing/uphill battle to keep the formula working.

What I've been leaning towards as a solution for this is to effectively move watchman out of homebrew core and manage it via our taps repo at https://github.com/facebook/homebrew-fb.

We could automatically update the tap as part of our weekly release automation and have it install the most recently successful macos binary build published to our releases section.

This feels like less work for everyone overall, and I have made a proof of concept of this approach in a personal project:
https://github.com/wez/wezterm/blob/master/.github/workflows/gen_macos_tag.yml#L85-L108 which updates a tab formula here: https://github.com/wez/homebrew-wezterm/blob/master/Formula/wezterm.rb

I haven't had an opportunity to sit down and make this happen so far, but if you're feeling motivated and have the bandwidth for it, we'd love to see you submit a couple of PRs for this.

What I think needs to happen:

Then we can look at deprecating the homebrew core formula. I think that this will result in something that is easier to maintain overall.

What do you think?

@isker
Copy link
Author

isker commented Aug 25, 2020

This sounds reasonable. Monorepos are The Way 😉.

I'm confused about what the collisions/opt part is for though. Won't homebrew link it straight into /usr/local/bin when installing anyway?

@wez
Copy link
Contributor

wez commented Aug 26, 2020

I'm confused about what the collisions/opt part is for though. Won't homebrew link it straight into /usr/local/bin when installing anyway?

By default, yes, but for example: at FB we have a site specific homebrew deployment set to deploy to /opt/homebrew and there's some other complicated chef stuff happening that manages /usr/local/bin and its permissions which makes it frustrating to use their own user specific stock homebrew. Some people choose to do that anyway, and others do a per-user install. If we hard code /usr/local/bin into the binaries then we make it difficult to co-exist with other configurations of homebrew, the current core brew formula or other distribution systems.

Outside of homebrew/mac, some users have commented that they'd appreciate having the linux binaries not deploy /usr/local so that they can do things like mount them via NFS more easily.

isker pushed a commit to isker/homebrew-fb that referenced this issue Aug 26, 2020
The formula in homebrew-core is moribund, and as per
facebook/watchman#851 (comment),
continuing to build watchman from source with brewed dependencies is
untenable.  So we add one that just distributes the released binaries here.
isker pushed a commit to isker/homebrew-fb that referenced this issue Aug 26, 2020
The formula in homebrew-core is moribund, and as per
facebook/watchman#851 (comment),
continuing to build watchman from source with brewed dependencies is
untenable.  So we add one that just distributes the released binaries here.

Eventually the url and sha will be updated automatically by the
watchman release process.
isker added a commit to isker/homebrew-fb that referenced this issue Aug 26, 2020
The formula in homebrew-core is moribund, and as per
facebook/watchman#851 (comment),
continuing to build watchman from source with brewed dependencies is
untenable.  So we add one that just distributes the released binaries here.

Eventually the url and sha will be updated automatically by the
watchman release process.
@isker
Copy link
Author

isker commented Aug 26, 2020

So, here's where my inexperience with native {builds,packaging,anything at all} fails me: why would the prefix get baked into the binaries at all? I thought that the prefix was just to tell make install where to copy the binaries.

I set up the formula at facebook/homebrew-fb#41.

isker added a commit to isker/homebrew-fb that referenced this issue Aug 26, 2020
The formula in homebrew-core is moribund, and as per
facebook/watchman#851 (comment),
continuing to build watchman from source with brewed dependencies is
untenable.  So we add one that just distributes the released binaries here.

Eventually the url and sha will be updated automatically by the
watchman release process.
isker added a commit to isker/homebrew-fb that referenced this issue Sep 3, 2020
The formula in homebrew-core is moribund, and as per
facebook/watchman#851 (comment),
continuing to build watchman from source with brewed dependencies is
untenable.  So we add one that just distributes the released binaries here.

Eventually the url and sha will be updated automatically by the
watchman release process.
isker added a commit to isker/homebrew-fb that referenced this issue Sep 3, 2020
The formula in homebrew-core is moribund, and as per
facebook/watchman#851 (comment),
continuing to build watchman from source with brewed dependencies is
untenable.  So we add one that just distributes the released binaries here.

Eventually the url and sha will be updated automatically by the
watchman release process.
KapJI pushed a commit to facebook/homebrew-fb that referenced this issue Feb 12, 2021
The formula in homebrew-core is moribund, and as per
facebook/watchman#851 (comment),
continuing to build watchman from source with brewed dependencies is
untenable.  So we add one that just distributes the released binaries here.

Eventually the url and sha will be updated automatically by the
watchman release process.
@kelset
Copy link

kelset commented Feb 18, 2021

hey folks, trying to catch up on this whole conversation; it sounds like, per @isker PR merged last week by @KapJI: facebook/homebrew-fb#41 now if someone wants to install watchman via homebrew and not get stuck on the 2017's 4.9.0 version they'd need to:

brew tap facebook/fb
brew install facebook/fb/watchman

correct?

The side problem I'm seeing about this is that @isker's PR "hardcoded" the version in the .rb file to v2020.08.17.00, which at this point is fairly old too (6+ months). Is there a way to update that automatically when a new version is out? Or is the expectation that someone will always have to manually go and override that line in the config?


PS: btw @isker thank you SO MUCH for deciding to drive this problem and make that PR 👏👏

@KapJI
Copy link
Member

KapJI commented Feb 18, 2021

As for now, it needs to be maintained manually. Feel free to send PR for the new version.

@isker
Copy link
Author

isker commented Feb 18, 2021 via email

@fanzeyi
Copy link
Member

fanzeyi commented Feb 18, 2021

I think I can set up the GitHub action to automatically send PR to the Facebook repo updating the version every release.

@KapJI
Copy link
Member

KapJI commented Feb 18, 2021

Buck has this implemented here: https://github.com/facebook/buck/blob/master/tools/release/platforms/homebrew.py
You can make something similar.

@kelset
Copy link

kelset commented Feb 24, 2021

that would be great @fanzeyi 👍 do you know if/when you'll be able to do it?

@fanzeyi
Copy link
Member

fanzeyi commented Feb 24, 2021

@kelset maybe later this week. Will update soon =)

@fanzeyi
Copy link
Member

fanzeyi commented Feb 27, 2021

Some update, I have a version working and it opened a PR here facebook/homebrew-fb#56 in my testing. I'm now waiting on our open source team for a bot token I can use to make these commits :)

@kelset
Copy link

kelset commented Mar 1, 2021

dope 👍 lmk once it works :) I'm planning to do a PR for the react-native docs once it's all set up :)

facebook-github-bot pushed a commit that referenced this issue Mar 6, 2021
Summary:
#851

 ---

In my local testing it generated facebook/homebrew-fb#56

Pull Request resolved: #896

Reviewed By: wez

Differential Revision: D26858965

Pulled By: fanzeyi

fbshipit-source-id: 4f53e292c9f00acdc3283627365a5a9786bcc1c8
@kelset
Copy link

kelset commented Apr 15, 2021

@fanzeyi any updates on this?

facebook-github-bot pushed a commit that referenced this issue Jun 2, 2024
Summary: Pull Request resolved: facebookincubator/zstrong#851

Reviewed By: markbt

Differential Revision: D58012216

fbshipit-source-id: 2143ba626421f7b6048e9e2ce3522aac614a8b72
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants