Cannot launch server using packwiz due to rate-limiting by Github API

[TheGlitch76]

Describe the problem

When using a Packwiz modpack, the server script makes an API request to Github each time it's launched:

curl -fsSL https://api.github.com/repos/packwiz/packwiz-installer-bootstrap/releases/latest

However, Github has a very low rate limit for unauthenticated users, so if the server gets stuck in e.g. a crash loop with restart: unless_stopped or similar, the Github API will start returning 403s (from what I can tell, for about an hour) for rate-limiting.
This causes the curl request to fail and the container stops trying to boot:

curl: (22) The requested URL returned error: 403

Ideally, the container would either:

• Remove use of the GH API so that the rate-limiting never gets hit
• Gracefully handle 403s from the GH API when a cached version of packwiz-installer-bootstrap is available

Container definition

version: "3"

services:
  mc:
    image: itzg/minecraft-server
    ports:
      - 25565:25565
    environment:
      EULA: "TRUE"
      TYPE: QUILT
      VERSION: 1.18.2
      PACKWIZ_URL: REMOVED PACK URL
      MEMORY: 7G
      USE_AIKAR_FLAGS: true
      DEBUG: true
    tty: true
    stdin_open: true
    restart: unless-stopped
    volumes:
    - ./minecraft-data:/data:Z

Container logs

+ export HOME=/data
+ HOME=/data
++ id -u
++ id -g
++ ls -lnd /data
+ log 'Running as uid=1000 gid=1000 with /data as '\''drwxrwxr-x. 11 1000 1000 4096 Aug 15 16:24 /data'\'''
+ local oldState
++ shopt -po xtrace
+ oldState='set -o xtrace'
+ shopt -u -o xtrace
[init] 2022-08-15 16:28:57+00:00 Running as uid=1000 gid=1000 with /data as 'drwxrwxr-x. 11 1000 1000 4096 Aug 15 16:24 /data'
+ '[' '!' -e /data/eula.txt ']'
+ isTrue false
+ local oldState
++ shopt -po xtrace
+ oldState='set -o xtrace'
+ shopt -u -o xtrace
+ return 1
+ isTrue false
+ local oldState
++ shopt -po xtrace
+ oldState='set -o xtrace'
+ shopt -u -o xtrace
+ return 1
+ [[ -n '' ]]
+ [[ -n '' ]]
+ fixJavaPath
+ which java
+ export VERSIONS_JSON=https://launchermeta.mojang.com/mc/game/version_manifest.json
+ VERSIONS_JSON=https://launchermeta.mojang.com/mc/game/version_manifest.json
+ case "X$VERSION" in
+ VANILLA_VERSION=1.18.2
+ export VANILLA_VERSION
++ get_major_version 1.18.2
++ version=1.18.2
++ echo 1.18.2
++ cut -d. -f 1-2
+ MAJOR_VANILLA_VERSION=1.18
+ export MAJOR_VANILLA_VERSION
+ log 'Resolved version given 1.18.2 into 1.18.2 and major version 1.18'
+ local oldState
++ shopt -po xtrace
+ oldState='set -o xtrace'
+ shopt -u -o xtrace
[init] 2022-08-15 16:28:57+00:00 Resolved version given 1.18.2 into 1.18.2 and major version 1.18
+ cd /data
+ export ORIGINAL_TYPE=QUILT
+ ORIGINAL_TYPE=QUILT
+ isTrue false
+ local oldState
++ shopt -po xtrace
+ oldState='set -o xtrace'
+ shopt -u -o xtrace
+ return 1
+ isTrue false
+ local oldState
++ shopt -po xtrace
+ oldState='set -o xtrace'
+ shopt -u -o xtrace
+ return 1
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ -n '' ]]
+ versionLessThan 1.7
+ mc-image-helper compare-versions 1.18.2 lt 1.7
+ log 'Resolving type given QUILT'
+ local oldState
++ shopt -po xtrace
+ oldState='set -o xtrace'
+ shopt -u -o xtrace
[init] 2022-08-15 16:28:58+00:00 Resolving type given QUILT
+ case "${TYPE^^}" in
+ exec /start-deployQuilt
+ [[ -n '' ]]
+ [[ -n '' ]]
+ [[ LATEST = LATEST ]]
+ log 'Checking Quilt Installer version information.'
+ local oldState
++ shopt -po xtrace
+ oldState='set -o xtrace'
+ shopt -u -o xtrace
[init] 2022-08-15 16:28:58+00:00 Checking Quilt Installer version information.
++ maven-metadata-release https://maven.quiltmc.org/repository/release/org/quiltmc/quilt-installer/maven-metadata.xml
+ QUILT_INSTALLER_VERSION=0.4.3
+ [[ LATEST = LATEST ]]
+ log 'Checking Quilt Loader version information.'
+ local oldState
++ shopt -po xtrace
+ oldState='set -o xtrace'
+ shopt -u -o xtrace
[init] 2022-08-15 16:28:58+00:00 Checking Quilt Loader version information.
++ maven-metadata-release https://maven.quiltmc.org/repository/release/org/quiltmc/quilt-loader/maven-metadata.xml
+ QUILT_LOADER_VERSION=0.17.3
+ export INSTALLER=quilt-installer-0.4.3.jar
+ INSTALLER=quilt-installer-0.4.3.jar
+ export SERVER=quilt-server-1.18.2-0.17.3-launch.jar
+ SERVER=quilt-server-1.18.2-0.17.3-launch.jar
+ export QUILT_INSTALLER_URL=https://maven.quiltmc.org/repository/release/org/quiltmc/quilt-installer/0.4.3/quilt-installer-0.4.3.jar
+ QUILT_INSTALLER_URL=https://maven.quiltmc.org/repository/release/org/quiltmc/quilt-installer/0.4.3/quilt-installer-0.4.3.jar
+ [[ ! -e quilt-server-1.18.2-0.17.3-launch.jar ]]
+ [[ ! -e quilt-server-1.18.2-0.17.3-launch.jar ]]
+ export FAMILY=FABRIC
+ FAMILY=FABRIC
+ exec /start-setupWorld
+ '[' QUILT = CURSEFORGE ']'
+ worldDest=/data/world
+ [[ -n '' ]]
+ exec /start-setupVanillaTweaks
+ out_dir=/data/world/datapacks
+ isTrue false
+ local oldState
++ shopt -po xtrace
+ oldState='set -o xtrace'
+ shopt -u -o xtrace
+ return 1
+ [[ -n '' ]]
+ [[ -n '' ]]
+ exec /start-setupForgeApiMods
+ isTrue false
+ local oldState
++ shopt -po xtrace
+ oldState='set -o xtrace'
+ shopt -u -o xtrace
+ return 1
+ '[' '' ']'
+ '[' '' ']'
+ exec /start-setupModpack
+ CURSE_URL_BASE=https://minecraft.curseforge.com/projects
+ isTrue false
+ local oldState
++ shopt -po xtrace
+ oldState='set -o xtrace'
+ shopt -u -o xtrace
+ return 1
+ [[ -n REMOVED PACK URL ]]
++ curl -fsSL https://api.github.com/repos/packwiz/packwiz-installer-bootstrap/releases/latest
curl: (22) The requested URL returned error: 403 
+ latestPackwiz=
exit code: 22

[itzg]

Using a cached installer bootstrap seems quite reasonable.

I'm open to suggestions about not calling GitHub in the first place.

In any case, a PR is very welcome if you want to take a shot at it.

[comp500]

I am working on setting up packwiz-installer-bootstrap (and packwiz-installer) to be published to my own Maven repository, which won't have any excessive ratelimits; are you able to handle parsing maven-metadata.xml or would an alternative API be preferred?

[itzg]

That would be great. Yes, I am reading maven metadata elsewhere, so quite doable.

[comp500]

I don't have quite enough time to finish the bootstrap changes now; but I noticed you're already handling updating the bootstrapper, so why not just let you launch packwiz-installer.jar directly?! So I added a main method to packwiz-installer that lets you do so, with a command referencing the main class directly like java -cp packwiz-installer.jar link.infra.packwiz.installer.Main. It isn't the default Main-Class and prints a message, to encourage users to use the bootstrapper if they aren't manually handling updates.

packwiz-installer is available at the release packwiz maven (https://maven.packwiz.infra.link/repository/release/) with the dist classifier to indicate the fatjar including dependencies; i.e. https://maven.packwiz.infra.link/repository/release/link/infra/packwiz/packwiz-installer/v0.5.4/packwiz-installer-v0.5.4-dist.jar

[itzg]

@comp500 FYI the CDN being used doesn't seem to fully process the If-Modified-Since header since it still responds with 200 and the file content.

[mc-image-helper] 21:03:15.293 DEBUG : Request: GET https://maven.packwiz.infra.link/repository/release//link/infra/packwiz/packwiz-installer/v0.5.4/packwiz-installer-v0.5.4-dist.jar with headers [If-Modified-Since: Fri, 2 Sep 2022 01:27:56 GMT, Accept-Encoding: gzip, x-gzip, deflate]
[mc-image-helper] 21:03:15.410 DEBUG : Response: status=200, reason=OK, headers=[Date: Fri, 02 Sep 2022 02:03:19 GMT, Content-Type: application/java-archive, Content-Length: 4371210, Connection: keep-alive, Server: BunnyCDN-IL-940, CDN-PullZone: 402451, CDN-Uid: bcc1cfaa-dbd6-4a99-a5d9-95d88c8e8a34, CDN-RequestCountryCode: US, Cache-Control: public, max-age=2592000, Last-Modified: Thu, 01 Sep 2022 01:47:25 GMT, CDN-StorageServer: DE-169, CDN-FileServer: 435, CDN-ProxyVer: 1.02, CDN-RequestPullSuccess: True, CDN-RequestPullCode: 206, CDN-CachedAt: 09/02/2022 01:21:08, CDN-EdgeStorageId: 940, CDN-Status: 200, CDN-RequestId: a6782f9f9feab434deb617b2ea451115, CDN-Cache: HIT, Accept-Ranges: bytes]