HTTP requests take a long time to process

[irsyadhanif]

I'm seeing an issue where HTTP requests can take a long time to process before a response is sent, when it should be almost instant. This usually manifests when a client on a different machine has performed a successful HTTP request prior to the stuck HTTP request on the first machine.

As an example, I have two clients on an airgapped network and a server running romt. On the first client, I am able to pull crates and access the server directory through a web browser with no perceived slowness. I then go on the second client and access the server directory through a web browser with no perceived slowness. When I go back to the first client and attempt to pull crates or access the server directory, it will hang on the request (anywhere from ~90 seconds to forever) before it responds. Additionally, while the first client is stuck after a request, if the second client performs a request the server will not only respond to the second client, but it will also respond to the first client with the data it requested.

I do not believe this to be an issue with our network. When I use wireshark, I can see that the client is successfully communicating with the server. The client sends the HTTP request and the server ACKs it before it hangs.

We currently have the romt server running on a Ubuntu machine and it's configured to serve over unencrypted HTTP. We haven't tried using nginx, but we will try that configuration to see if this manifests. If anything seems unclear, I can grab screenshots or a screen capture to demonstrate.

[drmikehenry]

If I'm understanding correctly, you are running romt serve on an Ubuntu host on your disconnected network, and using cargo fetch (or similar) from each of two hosts on this network in order to access the mirrored content.

romt serve is essentially a wrapper around python -m http.server that provides two additional main features: it sets up git-http-backend as a CGI process for serving crates.io-index, and it provides a bit of URL remapping for crates for certain cases. The actual HTTP-based serving is being provided by the underlying http.server code. There's not much custom romt-provided code here to go wrong, and Python's http.server is mature and in my opinion unlikely to be causing the issues you are seeing.

The problems you describe do sound network-related to me. One way to test this is to use python -m http.server instead of romt serve and see if you can trigger this kind of hanging via web browsing from both clients (or by using command-line tools such as curl, wget, etc.). If you still get hangs, the problem is at a lower level than romt.

If testing as above doesn't cause hangs in the same way as using cargo fetch on the clients, then you can try separating the serving of the crates.io-index Git repository from the crates themselves. To do this, clone the crates.io-index to one of the client machines, and on that machine change ~/.cargo/config to point to localhost, e.g.:

[source.crates-io]
registry = 'http://localhost:8000/git/crates.io-index'

Then run romt serve on this local machine for the sole purpose of serving the git/crates.io-index Git repository to this client (the other client will continue to use the server, or you can setup both clients for local Git serving). The repository's config file will still point to your Ubuntu server host such that individual .crate files will be served from the server while the index is served from the client. Test using cargo fetch and the like on both clients this way.

The network-related issues that come to my mind which have caused similar problems for me in the past have involved accidental duplication of client MAC addresses and/or IP addresses, causing the network to become confused regarding how to route back to each client. I'd suggest checking that all addresses are unique on your network as an easy step to rule out this kind of problem.

[drmikehenry]

One thing I forgot to mention: the URL remapping that romt serve does for .crate URLs is needed to handle the default URLs that cargo uses. When romt was first written, URL remapping was necessary to match cargo's expectations with the on-disk layout romt uses for crates. Since that time, cargo now supports a {prefix} marker to eliminate the need for this URL remapping (see https://github.com/rust-lang/cargo/blob/master/src/doc/src/reference/registries.md#index-format).

The URLs which cargo uses are based on the dl key in crates.io-index/config.json. By default, romt configures this file to:

  "dl": "http://SERVER:8000/crates/{crate}/{crate}-{version}.crate",
  "api": "http://SERVER:8000/"

To change the URL pattern to match romt's expectations, add {prefix} into the dl key as shown below:

  "dl": "http://SERVER:8000/crates/{prefix}/{crate}/{crate}-{version}.crate",
  "api": "http://SERVER:8000/"

Commit the config.json file into the working branch of crates.io-index so that python -m http.server will receive URLs that match the on-disk crates/ layout.

[irsyadhanif]

We're pretty sure our network isn't the issue - every client we've tested with has their own IP address and the MAC addresses aren't duplicated. We did a test with python3 -m http.server and we didn't see the hang issue. Setting up romt on the client machines to serve crates would be sub-optimal, since we expect to have more client machines hitting the crates repo in the future and we'd like to easily update the crates repo every so often.

We ended up using nginx and it looks like all clients are now able to hit the romt server with no hang. Thank you for the quick response though!

[drmikehenry]

Thanks for verifying that your network addresses are unique. I'm glad to hear that nginx is working for you. The idea of running romt serve on the client wasn't intended as a long-term solution; it was just for testing to help isolate where the hanging is occurring (whether on the serving of static .crate files or from serving the crates.io-index Git repository). Putting the Git repository on one client would mean that client would split the requests between localhost (for Git) and the Ubuntu server (for .crate files), which might aid in figuring out what's going on. For the test with python -m http.server on the server, this would allow using cargo on the client rather than just a web browser or curl et al., which might show whether some interaction with cargo helps trigger the hanging condition. Depending on the outcome of this kind of testing, there could be other tests to help narrow down the cause of the hanging.

I don't know how to replicate the issue you were seeing. I tried a three-node configuration similar to what you describe and ran cargo fetch from two client machines simultaneously without causing a hang (though no doubt you've tried more times than I have). Since you have a working solution, there may not be enough benefit for you to continue tracking down what's wrong in the romt serve situation. Maybe your recent results with python -m http.server point toward some weirdness with serving the Git repository, but since I can't trigger the hanging situation there's only guesswork involved here.

In any event, I'm glad to hear your system is up and running :-)

[drmikehenry]

Since things are working for you and I can't replicate the issues you were seeing, I'm closing this ticket. Feel free to re-open if more information comes to light.