x/build: add LUCI openbsd-ppc64 builder

[n2vi]

Following the instructions at Dashboard builders:

hostname openbsd-ppc64-n2vi

CSR is attached after renaming since Github doesn't seem to allow attaching with the name openbsd-ppc64-n2vi.csr you asked for.

[gopherbot]

Change https://go.dev/cl/534976 mentions this issue: main.star: add openbsd-ppc64, linux-riscv64, freebsd-riscv64 builders

[dmitshur]

Thanks. Here's the resulting certificate: openbsd-ppc64-n2vi-1697128325.cert.txt.

I've mailed CLs to define your new builder in LUCI and will comment once that's done.

[n2vi]

Thank you; I confirm that using the cert I get a plausible looking luci_machine_tokend/token.json.

[dmitshur]

Glad to hear!

Step 3 is complete (CL 534976 is submitted), so you should be able to proceed with the next steps. Please feel free to comment here if you see something unexpected, or run into a problem that the documentation doesn't cover. Thanks.

[n2vi]

I have not read the code yet to diagnose this; leaving assigned to me.

2023/10/13 18:29:39 Bootstrapping the swarming bot with certificate authentication
2023/10/13 18:29:39 retrieving the luci-machine-token from the token file
2023/10/13 18:29:39 Downloading the swarming bot
2023/10/13 18:29:39 Starting the swarming bot /home/swarming/.swarming/swarming_bot.zip
72354 2023-10-13 18:29:47.331 E: ts_mon monitoring is disabled because the endpoint provided is invalid or not supported:
72354 2023-10-13 18:29:48.890 E: Request to https://chromium-swarm.appspot.com/swarming/api/v1/bot/handshake failed with HTTP status code 403: 403 Client Error: Forbidden for url: https://chromium-swarm.appspot.com/swarming/api/v1/bot/handshake
72354 2023-10-13 18:29:48.891 E: Failed to contact for handshake, retrying in 0 sec...

[n2vi]

I don't see anything in the code or logs here that help me diagnose. It just looks like the server didn't like the token.json that had been refreshed just a minute before.

Maybe someone there can check server-side luci logs? Unable to reassign to dmitshur; hope someone there sees this.

[dmitshur]

Thanks for the update.

I recall there was a similar looking error in #61666 (comment). We'll take a look.

[n2vi]

In case it helps... I set both -token-file-path on the bootstrapswarm command line and also LUCI_MACHINE_TOKEN in the environment. The logs don't indicate any trouble reading the token.json file, though they're not very explicit.

I appreciate that there have been serious security flaws in the past from too-detailed error messages. But I'd venture that it is safe for luci to say more than "403".

I recognize I'm a guinea pig for the Go LUCI stuff, so happy to give you a login on t.n2vi.com if you would find it easier to debug directly or hop on a video call with screen sharing.

Finally, I recognize I'm a newcomer to Go Builders. So it could well be user error here.

[dmitshur]

Thanks for your patience as we work through this and smooth out the builder onboarding process.

I set both -token-file-path on the bootstrapswarm command line and also LUCI_MACHINE_TOKEN in the environment.

To confirm, are both of them set to the same value, which is the file path location of the token.json file? If you don't mind experimenting on your side, you can check if anything is different if you leave LUCI_MACHINE_TOKEN unset and instead rely on the default location for your OS (/var/lib/luci_machine_tokend/token.json I believe).

We'll keep looking into this on our side. Though next week we might be somewhat occupied by a team event, so please expect some delays. Thanks again.

[n2vi]

Yes, both are set to the same value /home/luci/luci_machine_tokend/token.json. (My OS doesn't have /var/lib and anyway not a fan of leaving cleartext credentials in obscure corners of the filesystem.)

This morning I've retried the same invocation of bootstrapswarm as before and don't get the 403 Client Error. So maybe there was just a transient issue.

Happy to set this effort on the shelf for a week or two; enjoy the team event!

[dmitshur]

CC @golang/release.

[n2vi]

Over the last week I tried swarm a few more times with no problems, so whatever issue I saw before indeed seems transient. I never saw swarm do any actual work, presumably because some server-side table is still pointing to my machine as in the old-builder state rather than new-builder. Fine by me.

I'll have limited ability to work on it from November 8 - 20, but happy to work on it during the next few days if you're waiting on me.