OSError: [Errno 7] Argumento list too long: '/bin/sh' during Firewall rules upgrades

[migldasilva]

Describe the bug
I'm working with an AzureTRE environment with around 25 workspaces. During a new workspace deployment, the Firewall upgrade failed with the following error:

OSError: [Errno 7] Argumento list too long: '/bin/sh' 

The full message can be seen in the attached screenshot.

After some debugging, I concluded that parameters sent to Porter, using --param network_rule_collections and --param rule_collections were too long. I mean, the Base64 encoded strings represent Rule collections and Application rule collections, were longer than the Linux image were able to handle.

This environment is based on AzureTRE 0.9.0, and Firewall shared service is version 1.1.7. However, comparing current Resource Processor version and the version we have deployed, the command seems to be based in the same approach; e.g., a single Porter upgrade command with all the parameters are created and launched.

Have you considered a different approach for updating Firewall rules?

I have tried to split the Porter upgrade command into 2 commands. One includes only --param network_rule_collections and the other includes only --param rule_collections. Preliminary tests went fine and I was able to unblock deployments, but I'm aware it's a mitigation.

Steps to reproduce

1. Deploy lots of workspaces and lots of workspace services so that around 300 rules are created
2. Firewall upgrade should fail due to too long list of parameters

Azure TRE release version (e.g. v0.14.0 or main):

Release version is v0.9.0

Deployed Azure TRE components - click the (i) in the UI:

[marrobi]

@migldasilva I've not seen this, but can see how it could be an issue. have you got some services with lots of rules?

Researching with copilot I've come up with:
This is the Linux execve limit (MAX_ARG_STRLEN = 128 KiB per argument, ARG_MAX ≈ 2 MiB total). In resource_processor/helpers/commands.py, build_porter_command base64-encodes complex parameters and appends each as a single CLI argument, so aggregated network_rule_collections / rule_collections eventually blow past the limit. Splitting the upgrade into two porter calls (your mitigation) helps short-term but a single param can still exceed 128 KiB on its own.

Two options to fix this properly

Option A — threshold-based: keep --param key=value for small values, switch to a Porter parameter set only when a value exceeds e.g. 100 KiB. Minimal change, but two code paths to maintain and a magic threshold to tune.

Option B (recommended) — always use a parameter set, sourced from env vars: drop --param entirely. For every parameter, set PORTER_PARAM_<NAME> in the subprocess environment and reference it from a generated parameter set (source: { env: PORTER_PARAM_<NAME> }). Then porter parameters apply <set>.json once, and run porter <action> ... --parameter-set <name>.

Option B is simpler (one code path), permanently removes ARG_MAX as a concern for all bundles, keeps large/secret values out of ps listings and log dumps of the failed command, and makes the per-run command line readable again.

Implementation notes for Option B

• Use a unique parameter-set name per run (e.g. tre-params-<installation_id>) so concurrent RP workers don't collide.
• Build the env dict alongside the parameter-set JSON and pass it into asyncio.create_subprocess_exec via config["porter_env"] — do not mutate the RP process's own environment, build a fresh dict per run so values are scoped to that subprocess and discarded when it exits.
• After the porter command completes (success or failure), delete the temp parameter-set file and porter parameters delete <name> so nothing persists in PORTER_HOME between runs.
• Existing base64 encoding of dict/list values stays — only the delivery mechanism to Porter changes.

@migldasilva if aligned, will get copilot to create a PR

[marrobi]

Should consider env var limits, maybe need to use files.

[marrobi]

Proposed fix: Porter parameter sets via temp file

Solution: Instead of passing parameters on the command line, the resource processor will:

1. Write a single temporary parameter set JSON file containing all parameters (inline values):

   {
     "name": "tre-params-<installation_id>",
     "parameters": [
       { "name": "param_name", "source": { "value": "..." } }
     ]
   }

2. Run porter parameters apply /tmp/tre-params-<id>.json
3. Run porter <action> ... --parameter-set tre-params-<id> (no --param flags)
4. Clean up the temp file and porter parameters delete in a finally block

Why this approach:

• Eliminates all kernel execve limits — file size is bounded only by disk
• Single code path (not threshold-based)
• No new scripts or infrastructure added to bundles
• No changes to workspace templates
• Keeps secrets out of ps output and error logs
• Uses unique installation_id per run to avoid concurrency issues
• Scales comfortably to 100+ workspaces (ultimate ceiling becomes Cosmos document size at ~2 MB ≈ 1,300 workspaces)

[migldasilva]

@marrobi Thank's for the proposals. Indeed, we have workspace templates with many rules. For the time being, splitting the porter commands is helping us, but will not be enough.

I'll try your proposals and come back to report it.

[migldasilva]

@marrobi I'm testing ParameterSet approach and I'd like to say that the parameter set is not loaded directly from a file, when porter upgrade is executed. First we apply the parameter set, and the upgrade command will load the applied set.

Essentially, before porter upgrade, we need porter parameter apply.

The Porter command to be created should look like this:

(...)
porter parameters apply <PARAMETER_SET_JSON_FILE> &&
porter upgrade (...) --parameter-set <PARAMETER_SET_NAME> ... &&
(...)

Placeholder <PARAMETER_SET_NAME> refers to the name given to the parameter set (it's set inside the JSON file).

Thus, I'd suggest adding the porter parameters apply command, and the PR should be great. :)

[marrobi]

@migldasilva thanks. Did you get the latest code? The review flagged that and added it.

        commands.append(["porter", "parameters", "apply", param_set_file])

[migldasilva]

@marrobi I followed the PR and given that we are using an older version, I couldn't check the latest code. I'm glad that review realized that we need porter parameters apply.

I'm able now to update Firewall policies without problems. Even using a different version, the solution you have proposed, is working fine. Many thanks.