Unable to Connect to Tigerbeetle in Codespaces

[A-Mozeak]

Background

We have had successful runs of the Rafiki test suite (which is great) on our Virtualbox VM running Ubuntu 21.04 (to avoid security concerns running Docker with root privileges), but are not able to run the backend portion of the test suite inside of a GitHub Codespaces environment (which we are trying to run for various reasons including portability and security). The rates tests all run just fine 👍🏾.

The main issue seems to be that Tigerbeetle isn't able to connect to an actively listening replica (at port 3004). It could be that there is something missing from the default Codespaces environment that we need to add to our dev container (see Specifications).

Expected Behavior

Running yarn workspace backend test successfully connects to Tigerbeetle and passes (or fails) tests appropriately.

Actual Behavior

Running yarn workspace backend test fails to connect to Tigerbeetle and does not run the actual test logic.

Terminal output is:

error(message_bus): error connecting to replica 0: error.Unexpected
error(message_bus): error closing connection to tigerbeetle.src.message_bus.union:392:19{ .replica = 0 }: error.Unexpected

Tigerbeetle container output is:

> Executing task: docker logs --tail 1000 -f 43d4f1d1938a1fc684fa77d36fe3eff89966e211ce5cae0b0945f71f0b5962ad <

info(storage): opening "cluster_0000000001_replica_000.tigerbeetle"...
info: cluster=1 replica=0: listening on 0.0.0.0:3004
error(message_bus): accept failed: error.SocketNotListening

Terminal will be reused by tasks, press any key to close it.

I've looked into tigerbeetle/src/message_bus.zig to try to make sense of the situation but I am not familiar enough with the architecture to get a sense of what to do next.

Steps to Reproduce the Problem

1. Connect to a GitHub Codespace.
   • We are using VSCode locally to remotely connect to the codespace, but the same goes for a browser-based VSCode instance.
2. Add the rafiki repository as a submodule to the Codespace repository.
   • git submodule add https://github.com/interledger/rafiki.git
   • Probably worthwhile to check if the same effects occur if Rafiki is not a submodule but instead the active repo, though on the Virtualbox VM the tests ran fine in the submodule as well.
3. Run yarn build at the root of the repo.
4. Build the Docker container for the backend using yarn docker build backend -t rafiki-backend.
5. Run yarn workspace backend test.

Specifications

GitHub Codespaces Default Linux Container

[wilsonianb]

Am I understanding correctly that you are trying to run the Rafiki backend tests against a running Tigerbeetle Docker container?
If so, you will need to specify TIGERBEETLE_REPLICA_ADDRESSES. Otherwise, Rafiki starts a Tigerbeetle container when setting up the backend tests:
https://github.com/interledger/rafiki/blob/main/packages/backend/jest.setup.js#L67

If you opt to use the Rafiki managed Tigerbeetle container, you'll need to configure Docker to run as a non-root user.

[A-Mozeak]

Thanks for the reply @wilsonianb! I am trying to run the tests using the Rafiki managed Tigerbeetle container, and as far as I can tell, Docker is configured to run as a non-root user. Here's the output of the groups command showing the docker group.

codespace ssh docker golang rustlang sdkman nvm rvm pipx

And docker run hello-world works without the sudo command. This is the same setup I have on Virtualbox, where the tests run fine, so I don't think it's related to that part of the configuration.

I will try running my own Tigerbeetle container and specifying the TIGERBEETLE_REPLICA_ADDRESSES. Do you have any other thoughts on why it might say info: cluster=1 replica=0: listening on 0.0.0.0:3004 and then error(message_bus): accept failed: error.SocketNotListening right after?

[A-Mozeak]

No dice running Tigerbeetle manually either. I think this may actually have to do with this Codespaces issue, so I'm looking for a fix in that direction. I'll let you know how it goes!

[jorangreef]

Do you have any other thoughts on why it might say info: cluster=1 replica=0: listening on 0.0.0.0:3004 and then error(message_bus): accept failed: error.SocketNotListening right after?

Hi @A-Mozeak ! Thanks for the report. It looks like you are running a Linux kernel older than 5.5. TigerBeetle requires io_uring and the accept() syscall for io_uring only shipped in 5.5. That's why you're seeing accept failed. Under the hood, io_uring is returning E_INVAL for the accept syscall because the kernel is likely too old.

@kprotty is already working on an assertion to make this easier to catch in future.

[A-Mozeak]

@jorangreef Thank you, looking into this.

[A-Mozeak]

@jorangreef it looks like this is something that GitHub has to address on their end, since the base layer host for Codespaces is not user configurable.

We've changed our strategy and are working from our own cloud hosted Linux box, so I'm going to close this issue. Thanks to you and @wilsonianb for the help.