Using a Docker Cell From Rails Running on macOS

[sandstrom]

Hello again 😄 Trying to integrate Hotcell and ran into something that made me pause.

We run Rails directly on macOS and use Docker Compose for supporting services. We’d like to run Hotcell in Docker too, so developers don’t need to install libvips or other processing dependencies on their Macs.

The file-descriptor protocol makes that impossible today, since descriptors cannot cross into Docker’s Linux VM. The dcoumented options are to run the cell natively or move the application into Docker. Running the cell natively means installing the dependencies we’d hoped to keep mostly inside its image (apart for when working on hotcell itself).

Keeping them in Docker would let developers use the same processing libraries and versions as production without maintaining a separate macOS installation. It would also fit our existing workflow, where Rails runs locally for debugging and supporting services run in containers.

Would you consider supporting another transport for development?

A few possibilities come to mind:

1. A small HTTP relay running alongside the hotcell server could receive file bytes and operation arguments from macOS, create temporary files inside Linux, and call the cell through the existing descriptor protocol.

2. A helper invoked through docker exec could do similar work using stdin/stdout, avoiding a published HTTP endpoint.

3. Mount a dedicated exchange directory into Docker. The macOS client writes inputs there and sends a small request containing operation arguments and relative filenames. A Linux helper opens those files and forwards descriptors to the hotcell server.

These involve copying or staging files across the VM boundary. They wouldn’t have the same properties as the production transport, but that seems like a reasonable tradeoff for development.

There’s already a transport abstraction in the client. Perhaps this could be an optional implementation, or a documented extension point that lets teams build one without depending on internal APIs.

We’d like to keep operation code and application calls the same, with the transport chosen through development configuration.

Let me konw what you think!

[flavorjones]

Thanks for writing this up, @sandstrom. Good idea! Hot Cell was built to be extensible, so you can try this today without changing Hot Cell.

HotCell.register accepts a transport: option (cell.rb). A transport is an object that responds to call(cell, line, descriptors, socket:, timeout:) and returns a HotCell::Response. The client passes the following arguments:

• line: the request, already serialized.
• descriptors: HotCell::Input and HotCell::Output wrappers. Each one's io is the caller's open file.
• socket: the path to the cell's work.sock or control.sock. The client sends describe and metrics through the same call, with no descriptors.

The default transport, HotCell::Transport::Socket, sends the line and the descriptors over the Unix socket.

To call a containerized cell from Rails on macOS through an HTTP relay, do the following:

1. Run a relay in its own container beside the cell. The relay receives the request line and the input bytes from your app, stages them as files, calls the cell, and sends back the response and the output bytes. Leave the cell container's flags as they are, including --network none.

2. Connect the relay to the cell:

   • Mount the cell's socket volume in the relay's container, and add --group-add 10001.
   • Give each staged file group 10001, with mode 0640 for inputs and 0620 for outputs, as HotCell::Client does.
   • Wrap the files in HotCell::Input and HotCell::Output, and pass them to HotCell::Transport::Socket#call. Serialize the result with response.to_line.

3. In the Rails app, write a transport whose call sends the request to the relay, writes the returned bytes into each output's io, and returns HotCell::Response.parse of the returned line.

4. Register the cell with your transport in development:

   HotCell.register "active_storage",
     dir: "/run/hotcell/active_storage",
     transport: RelayTransport.new("http://127.0.0.1:9292")

   Set dir: to any path. The client treats a cell without a directory as disabled. On macOS, leave HotCell.group unset.

Your operation code and call sites don't change.

transport: isn't in the docs yet. I'd be happy to document it and its contract as part of the public API to support extensions like this. Once you have an implementation that you're using and happy with, let's talk about whether any of it belongs in Hot Cell itself, or whether it works better as a separate gem that you maintain.

[sandstrom]

Hello @flavorjones!

I saw transport: when looking around, but since it was undocumented didn't know if it was private or not. Great to hear it's public.

Before I start working, is the HTTP relay the approach you'd be most likely to upstream? Or is another approach something you'd prefer?

I understand you cannot give any guarantees now, but I'm pretty agnostic on the approach myself, so if I'm building something, I'd rather build whatever has the highest likelihood of getting into hotcell, since that would make it useful to people besides myself.

macOS is a pretty popular developer platform, so I'll imagine others will run into this as well.

[flavorjones]

@sandstrom I don't have opinions here, and because I'm not a MacOS user and don't have this problem, I want to avoid developing opinions. I recommend that you build what works best for you on top of Hot Cell, so we learn what extension points are missing or what accommodations Hot Cell could be making so the task is easy (or easier). Then I'm happy to make or adopt those changes to make Hot Cell extensible.

But to be absolutely clear, I would prefer if the bulk of this solution is NOT in Hot Cell itself, but was maintained my you and other MacOS users as a separate dev-only gem on top of those extension points.

[sandstrom]

Got it!

I'll investigate and try some things out. Will report back when I have something worthwhile to show (or more questions).

Thanks for the input on desired direction! 🙏🏻