Fix issues with v2 config parser#7249
Conversation
ConfigParserV2 overwrote its declaredProfiles set on every parse() call, so when ConfigBuilder reuses a single parser instance to parse multiple config files, only the profiles from the last file parsed were retained. A profile declared in an earlier file (e.g. the pipeline nextflow.config) was then reported as "Unknown configuration profile" when a later file (e.g. a launch-directory config) declared none. Accumulate declared profile names across parse() calls instead, matching the behaviour of the v1 parser. Already fixed on the default branch by the larger nextflow-io#6643 ("enable v2 syntax parser by default"); this is the minimal extraction for the 25.10.x line. Signed-off-by: Rob Syme <rob.syme@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Heya Ben. No, this PR does not fix #6623. They're separate bugs with separate fixes. Both bugs were fixed together in #6643, but by two independent code changes - my PR here only picks out one of those fixes. The two fixes are: Profile accumulation (the one this PR backports) - my customer bug Params from includeConfig (issue #6623) How about I adjust this PR to backport both fixes? |
When a config file assigns a param and then pulls in another file with includeConfig, ConfigParserV2 passed the already-accumulated config params to the include parser via setParams(). Those params were treated as CLI overrides, so ConfigDsl.assign() skipped the matching assignments in the included file and the parent value survived. A param declared with a default in the pipeline nextflow.config (e.g. `params.reads = null`) could therefore never be set by an included profile config, silently keeping the default instead of the included value. Separate the two kinds of params: cliParams (real `--param` overrides, which must win) and configParams (values accumulated from config files, passed to includes for merging only). Only cliParams gate assignment in ConfigDsl.assign(), so an included file can still set a param declared earlier while a genuine CLI override continues to take precedence. This is a distinct bug from the profile-accumulation fix in the previous commit; both were fixed on the default branch by the larger nextflow-io#6643 and are extracted here for the 25.10.x line. Fixes nextflow-io#6623 Signed-off-by: Rob Syme <rob.syme@gmail.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Problem
On the 25.10.x line, the strict (v2) config parser aborts a run at config-build time with
Unknown configuration profile: '<name>'whenever theprofiles {}block lives in a config file that is not the last one parsed. The legacy (v1) parser handles the same configuration fine.This surfaces on Seqera Platform launches in particular: Platform writes a config into the launch directory (a
tower {}scope, no profiles) which is parsed after the pipeline's ownnextflow.config, so every profile-using pipeline fails under the v2 parser. It reproduces with plainnextflow config/nextflow runagainst two config files;includeConfigis not involved.Root cause
ConfigBuildercreates a singleConfigParserinstance and callsparse()for each config file in order, then validates the requested-profileagainstparser.getDeclaredProfiles()once at the end.ConfigParserV2reassigned itsdeclaredProfilesfield on everyparse()call (declaredProfiles = script.getDeclaredProfiles()), so the field ended up holding only the profiles from the last file parsed. The v1 parser keeps afinalset and adds to it, accumulating across all files.Fix
Initialise
declaredProfilesto an empty set and accumulate withaddAll(...)acrossparse()calls, matching the v1 behaviour. Two lines.This is already fixed on the default branch as part of the larger #6643 ("enable v2 syntax parser by default"), which can't be cherry-picked cleanly onto 25.10.x; this PR is the minimal extraction of just the profile-accumulation fix.
Verification
should accumulate declared profiles across multiple config filesreuses one parser across twoparse()calls (the second declaring no profiles) and asserts the earlier file's profiles are retained.ConfigParserV2Testis unchanged and green.Scope / risk
The only non-test consumer of
getDeclaredProfiles()isConfigBuilder'scheckValidProfilegate, so the blast radius is limited to profile validation. The v2 parser is opt-in on 25.10.x (NXF_SYNTAX_PARSER=v2).