very slow builds through BSP on linux since v1.1.0

[ehigham]

Hi all,
I recently updated mill from v1.0.6 to v1.1.0 and noticed that builds via the bsp in intellij were
taking around 10x longer, with almost zero cpu usage. I tried later versions to the same effect.

The slowdown only manifests via bsp. Command line builds are roughly equivalent.
I've attached various mill-chrome-profile.jsons for command line and bsp build between these two versions.

Let me know any other details that you might need.
Thanks for your help in advance!

v1_0_6-mill-chrome-profile.json
v1_1_0-mill-bsp-chrome-profile.json
v1_1_0-mill-chrome-profile.json

[lihaoyi]

@ehigham could you also take a jprofiler or jfr profile of the MillDaemonMain and MillBspMain processes during the build to help identify sny hotspots?

[ehigham]

Here's a jfr tar.gz of the bsp build for 1.1.0:
v1_1_0-mill-bsp-main.jfr.tar.gz

[ehigham]

and daemon main for 1.0.6:
v1_0_6-mill-daemon-main.jfr.tar.gz

[ehigham]

v1_1_0-mill-daemon-main.jfr.tar.gz

[ehigham]

fwiw, I outsourced thinking to claude-opus-4-6 with max effort. See below for the output. I hope that maybe this helps point in the right direction, but I have little confidence in these systems atm.

  Findings

  The regression is BSP-specific

  CLI builds are equivalent across versions. The regression only manifests in BSP mode on v1.1.0:

  ┌────────────────────────┬────────────┬────────────┬────────────┐
  │                        │ v1.0.6 CLI │ v1.1.0 CLI │ v1.1.0 BSP │
  ├────────────────────────┼────────────┼────────────┼────────────┤
  │ Duration               │       144s │       139s │       959s │
  ├────────────────────────┼────────────┼────────────┼────────────┤
  │ Localhost socket reads │         47 │         42 │     21,728 │
  ├────────────────────────┼────────────┼────────────┼────────────┤
  │ Total socket wait      │       1.1s │       1.2s │       886s │
  └────────────────────────┴────────────┴────────────┴────────────┘

  The bottleneck: BSP makes 500x more RPC calls to the zinc subprocess

  In BSP mode on v1.1.0, the SubprocessZincApi → MillRpcClient.awaitForResponse → MillRpcWireTransport.read() code path makes
  21,728 socket reads to the zinc subprocess on localhost, compared to ~42 in CLI mode on the same version. Something in the
  v1.1.0 BSP code path queries the zinc subprocess per-module or per-target in a way the CLI path does not.

  Each RPC call hits a 40ms TCP penalty on Linux

  99.8% of those socket reads block for exactly 40ms:

      10 ms: 12
      15 ms: 8
      20 ms: 10
      25 ms: 9
      30 ms: 3
      35 ms: 4
      40 ms: 21694    <-- 99.8%
      45 ms: 1

  40ms is Linux's TCP_DELACK_MIN (always HZ/25 = 40ms). The interaction:

  1. The zinc subprocess writes a response in multiple small TCP segments
  2. The first segment is sent immediately
  3. The client's TCP stack delays the ACK for up to 40ms
  4. The subprocess writes the next segment, but Nagle's algorithm holds it (unacknowledged data in flight)
  5. After 40ms, the delayed ACK is sent, Nagle releases the next segment

  21,694 × 40ms = ~868 seconds of pure idle waiting on a TCP timer.

  Why this doesn't reproduce on macOS

  macOS defaults to net.inet.tcp.delayed_ack=3 — an adaptive algorithm that detects interactive traffic patterns and sends
  ACKs much faster than Linux's fixed 40ms floor. The same RPC calls complete in sub-millisecond on macOS loopback, masking
  the chattiness.

  Suggested fixes

  1. Root cause: Investigate why the v1.1.0 BSP code path makes ~500x more zinc subprocess RPC calls than the CLI path
  (~21,700 vs ~42 for the same build)
  2. TCP fix: Set TCP_NODELAY on the RPC socket (e.g. in ServerLauncher.runWithConnection or MillRpcWireTransport) to
  eliminate the 40ms penalty per round-trip:
  socket.setTcpNoDelay(true);
  2. This alone would drop the total socket wait from ~886s to ~10s, but the 500x call count disparity is still worth
  investigating.

[lefou]

The higher RCP message count in BSP mode could be caused by the events the BSP protocol sends to report compile progress.