feat: Add scie-based single binary distributions

[achimnol]

The following command will build single-binary self-contained self-bootstrapping executables under dist/ for the current platform.

pants --tag='-wheel' package ::

> ls -lh dist
total 409M
-rwxr-xr-x 1 joongi joongi 133M Nov  1 23:48 backendai-agent-linux-aarch64
-rwxr-xr-x 1 joongi joongi  44M Nov  1 23:48 backendai-agent-watcher-linux-aarch64
-rwxr-xr-x 1 joongi joongi  48M Nov  1 23:48 backendai-client-linux-aarch64
-rwxr-xr-x 1 joongi joongi  57M Nov  1 23:48 backendai-manager-linux-aarch64
-rwxr-xr-x 1 joongi joongi  52M Nov  1 23:48 backendai-storage-proxy-linux-aarch64
-rwxr-xr-x 1 joongi joongi  78M Nov  1 23:48 backendai-webserver-linux-aarch64

You could execute each service like:

./backendai-manager-{platform} mgr start-server
./backendai-agent-{platform} ag start-server
./backendai-storage-proxy-{platform} storage start-server
./backendai-webserver-{platform} web start-server
./backendai-agent-watcher-{platform}  # this is a direct wrapper of watcher.py

Note
The redundancy of package names in the executable (e.g., -manager) and the sub-command prefix (e.g., mgr) is inevitable because we need to be able to distinguish different click.Group providers when installing all packages in a single venv or developing in a mono-repo.
This pex+scie based packaging makes independent executables where the only required internal dependencies are included but all executables share the same unified entrypoint from ai.backend.cli.

You could execute the client SDK's CLI commands like:

./backend.ai-client-{platform} run ...

Hacking around the scie distributions

You could directly run the unzipped standalone Python build and the pex distribution after the first run like:

$ .cache/nce/3e26a672df17708c4dc928475a5974c3fb3a34a9b45c65fb4bd1e50504cc84ec/cpython-3.11.6+20231002-aarch64-unknown-linux-gnu-install_only.tar.gz/python/bin/python3 .cache/nce/a179cb83d7f37fb3ac1e1998712a8585937c7f73bce851ad555e2d69e0f966f5/src.ai.backend.install/pex.pex

To check out the hash digest values, you could run:

$ SCIE=inspect backendai-install-linux-aarch64
{
  "scie": {
    "lift": {
      "name": "backendai-install",
      "description": "",
      "files": [
        {
          "name": "ptex-linux-aarch64",
          "key": "ptex",
          "size": 3297184,
          "hash": "b71bea2deda31242e8d5d8749d173f4fe166182e8ebbef3ef605da06320bf24a",
          "type": "blob",
          "executable": true
        },
        {
          "name": "cpython-3.11.6+20231002-aarch64-unknown-linux-gnu-install_only.tar.gz",
          "key": "cpython",
          "size": 25547472,
          "hash": "3e26a672df17708c4dc928475a5974c3fb3a34a9b45c65fb4bd1e50504cc84ec",
          "type": "tar.gz",
          "source": "fetch"
        },
        {
          "name": "src.ai.backend.install/pex.pex",
          "size": 25165252,
          "hash": "a179cb83d7f37fb3ac1e1998712a8585937c7f73bce851ad555e2d69e0f966f5",
          "type": "blob",
          "executable": true
        }
      ],
      "boot": {
        "commands": {
          "": {
            "exe": "{cpython}/python/bin/python3.11",
            "args": [
              "{src.ai.backend.install/pex.pex}"
            ]
          }
        },
        "bindings": {
          "fetch": {
            "exe": "{ptex}",
            "args": [
              "{scie.lift}"
            ],
            "description": "Fetch files not present in the scie"
          }
        }
      }
    },
    "jump": {
      "size": 1643256,
      "version": "0.13.2"
    }
  },
  "ptex": {
    "cpython-3.11.6+20231002-aarch64-unknown-linux-gnu-install_only.tar.gz": "https://github.com/indygreg/python-build-standalone/releases/download/20231002/cpython-3.11.6%2B20231002-aarch64-unknown-linux-gnu-install_only.tar.gz"
  }
}

Checklist: (if applicable)

• Milestone metadata specifying the target backport version
• Mention to the original issue
• Documentation