Updated build instructions? (M1, Xcode 13.2.1, Dec 2021)

[luckman212]

In my quest to track down #3035 one thing I wanted to try was to compile on the latest Xcode / OS. So I sat down and tried to do what I've done in the past:

1. git clone https://github.com/Hammerspoon/hammerspoon.git
2. open Hammerspoon.xcworkspace
3. update signing info
4. build

That used to work fine for me. Not anymore... 🤔

Ok, I see there's a new build system. So first thing I tried was just simply

$ ./scripts/build.sh build

Complained about missing deps (pygments, etc). Ok, let's try

$ ./scripts/build.sh installdeps

Whoops... it wants to install a bunch of Python packages using the system Python (/usr/bin/python3) - yikes. And errors out:

ERROR: Could not find an activated virtualenv (required).
ERROR: Unable to install Python dependencies

Ok, so I have a working virtualenv setup and Python 3.9 via Homebrew, so:

$ mkvirtualenv hammerspoon

And then a few quick edits to the build scripts to use the venv flavors of pip and python:

• scripts/libbuild.sh:

216: pip3 install -r "${HAMMERSPOON_HOME}/requirements.txt" || fail "Unable to install Python dependencies"
599: pkg_resources.require(dependencies)" | python3

• scripts/docs/bin/build_docs.py (shebang):

#!/usr/bin/env python3

...Then:

$ ./scripts/build.sh installdeps
...[snip]...
Ruby packages...
WARNING:  You don't have /Users/luke/.gem/ruby/2.6.0/bin in your PATH,
	  gem executables will not run.

Hmm, ok:

$ PATH=/Users/luke/.gem/ruby/2.6.0/bin:$PATH
$ ./scripts/build.sh clean
$ ./scripts/build.sh build

(wall of warnings.........)

...followed by:

The following build commands failed:
	PhaseScriptExecution Compile\ docs.json /Users/luke/Downloads/hammerspoon/Build/Intermediates.noindex/ArchiveIntermediates/Hammerspoon/IntermediateBuildFilesPath/Hammerspoon.build/Debug/Hammerspoon.build/Script-4FC09F7A19F99E48007542A6.sh (in target 'Hammerspoon' from project 'Hammerspoon')
(1 failure)

So, something's wrong with building the docs, ehhh well I don't really need to build those anyway. Let me try the -d (debug) and -l flag (lint only, don't build docs):

./scripts/build.sh build -d -l

At this point I am ready to throw in the towel. Probably I'm missing something obvious but...
Please help 😐 @cmsj @asmagill

[cmsj]

Hey @luckman212 , sorry this has been such a miserable experience.
FWIW the docs.json file is required for the build because we ship it in the app bundle to allow HS' internal help tools (eg hs.help()) to have all the metadata they need.

I would like our git repo to build directly from a fresh clone on a machine that has never built Hammerspoon before, and there are some options we could explore:

• Separate the docs.json extraction/generation out into a Swift tool that can be built without external dependencies
• Figure out a GitHub workflow that ensures an updated docs.json file is committed to the repo any time we change docstrings

I'm curious why you got an error about a missing venv - I explicitly didn't use a venv for any of the dev/testing of the new build scripts, and the system Python3/pip3 are used explicitly to reduce the fragility of building in GitHub CI vs building on my machine.

I'll for sure extend the installdeps command to take an argument if it's preparing for a full release - the sole Ruby dependency is literally just a Twitter command line client used to tweet that a new release is out, so that's not required for a clone&build on a third party machine.

For the main problem here though, generating docs.json I suspect the GitHub Actions workflow would be fragile and confusing, since it would be adding commits that I would need to pull down during a full release build, so probably the way to go is to rewrite the docstrings extraction/collation into a Swift binary (which will also need to be made available somehow to the Spoons repo which also needs to generate API docs).

I realise this isn't a quick answer that gets you building (I'm away from my machine so can't easily experiment and get you going), but I figured it was worth giving some context and thinking out-loud about how we could simplify life for everyone by reducing our dependence on external tooling even further.

[cmsj]

Ah, I guess you have PIP_REQUIRE_VIRTUALENV=true in your environment, which would trigger the venv error.

[luckman212]

Thanks @cmsj for the details, I never even knew about hs.help so that's neat.

As I was typing this reply I see you correctly guessed that I have PIP_REQUIRE_VIRTUALENV set in my environment. I still struggle to manage multiple Python installs and find it to be something of a mess on macOS (relevant xkcd).

Anything else I can do or provide details wise to help make this better I am happy to.

[cmsj]

Absolutely agree that Python on macOS is a disaster, which is partly why I just bluntly hardcode the system Python/Pip, so at least I know we're always using the same environment.

FWIW I've started work on porting the docstrings extraction code to Swift. There's quite a bit to do though, and then it'll need extensive testing to try and avoid regressions.

if you need to be doing builds now, probably the quickest option would be disabling that PIP_REQUIRE variable, if you can live with the system Python installing a few modules.

[luckman212]

Got it. I was able to get further with unset PIP_REQUIRE_VIRTUALENV but am now stuck with this error about No "teamID" specified (I don't have a paid developer account) - I opened the xcproj in Xcode and set all the signing to "sign to run locally" with my personal appleID account but maybe I need to pass in some other flags to xcodebuild...?

(...snip)
[Hammerspoon] Running script Update version/build numbers
[Hammerspoon] Copying hs.man
[Hammerspoon] Copying sample.lp
[Hammerspoon] Copying search.lp
[Hammerspoon] Copying module.lp
[Hammerspoon] Copying gfm.lp
[Hammerspoon] Copying index.md
[Hammerspoon] Copying init.lua
[Hammerspoon] Copying index.lp
[Hammerspoon] Copying form.lp
[Hammerspoon] Copying common.lp
[Hammerspoon] Copying index.md
[Hammerspoon] Copying templates.md
[Hammerspoon] Copying functions.md
Signing Hammerspoon.app (in target 'Hammerspoon' from project 'Hammerspoon')
[Hammerspoon] Touching Hammerspoon.app
Archive Succeeded
+ '[' archive == archive ']'
+ xcodebuild -exportArchive -archivePath /Users/luke/Downloads/hammerspoon/build/Hammerspoon.app.xcarchive -exportOptionsPlist 'Hammerspoon/Build Configs/Archive-Export-Options.plist' -exportPath /Users/luke/Downloads/hammerspoon/build
objc[42166]: Class AMSupportURLConnectionDelegate is implemented in both /usr/lib/libauthinstall.dylib (0x1f9cf6b90) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x103a0c2c8). One of the two will be used. Which one is undefined.
objc[42166]: Class AMSupportURLSession is implemented in both /usr/lib/libauthinstall.dylib (0x1f9cf6be0) and /Library/Apple/System/Library/PrivateFrameworks/MobileDevice.framework/Versions/A/MobileDevice (0x103a0c318). One of the two will be used. Which one is undefined.
2021-12-23 01:47:25.442 xcodebuild[42166:195095] [MT] IDEDistribution: -[IDEDistributionLogging _createLoggingBundleAtPath:]: Created bundle at path "/var/folders/n1/kwxhk1g96vl592mftk5lwntc0000gn/T/Hammerspoon_2021-12-23_01-47-25.442.xcdistributionlogs".
error: exportArchive: No "teamID" specified and no team ID found in the archive

Error Domain=IDEFoundationErrorDomain Code=1 "No "teamID" specified and no team ID found in the archive" UserInfo={NSLocalizedDescription=No "teamID" specified and no team ID found in the archive}

** EXPORT FAILED **

[luckman212]

@cmsj I figured out a way to build successfully! 🚀 In case you're curious, here are the steps I ended up with:

1. install prerequisites (put the last 2 in your $PATH somewhere and chmod +x them):

• rg (ripgrep) brew install ripgrep
• jo (create json objects) brew install jo
• rgr (quick search & replace using ripgrep)
• devteamid (grabs dev teamID)

2. clone repo, run installdeps:

git clone https://github.com/Hammerspoon/hammerspoon.git
cd hammerspoon
unset PIP_REQUIRE_VIRTUALENV
./scripts/build.sh installdeps

3. replace DEVELOPMENT_TEAM and fix xcodebuild config plist to use it:

rgr VQCYSNZB89 $(devteamid.sh)
jo destination=export method=development teamID=$(devteamid.sh) | plutil -convert xml1 -o 'Hammerspoon/Build Configs/Archive-Export-Options.plist' -- -

4. build:

./scripts/build.sh build -d

5. sip coffee ☕, and...

bad news: even this freshly-minted build is still having the #3035 issue. Still stuck on what could possibly be causing that!

[cmsj]

Ok, so that DEVELOPMENT_TEAM thing is a legitimate issue I need to take care of.

As for the docs.json generation, @randomeizer have at this point implemented three different docstrings parsers and I think at this point we need to have a conversation about them.

In short, I don't really like any of them. It feels to me like we're forcing Swift to do something that it's not very well suited to, and in the case of two of the three implementations (both using different featuresets of swift-parsing) I think they are horribly fragile and depend very much on a third party Swift Package that is maintained more as an educational tool than anything else.

@randomeizer I'd be curious to know your thoughts, but atm I'm not at all convinced that it's worth pursuing that idea further.

In the meantime, I've rejigged build.sh and libbuild.sh so that they shouldn't actually force requirements.txt to be satisfied in the basic case where docs.json is required by the build, since none of the dependencies listed as requirements are actually required to generate the JSON output. This ought to get us back to a point of being able to clear that part of the Xcode build without any prior effort.

As for the DEVELOPMENT_TEAM I think we should be able to solve that with an out-of-tree optional include to override a default value, which I think should be literally just -.