Skip to content

fix: do not cap <IntRangeExpr> expansion at the list-form limit - #327

Merged
leongdl merged 1 commit into
mainlinefrom
fix/range-expr-expansion-not-capped
Aug 14, 2026
Merged

fix: do not cap <IntRangeExpr> expansion at the list-form limit#327
leongdl merged 1 commit into
mainlinefrom
fix/range-expr-expansion-not-capped

Conversation

@leongdl

@leongdl leongdl commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Problem

0.11.1 began enforcing a 1024-element cap on the expansion of an
<IntRangeExpr> task parameter range. §3.4 does not specify that cap for that
form.

The 1024 element limit is stated for the list forms only:

  • §3.4.1.1 item 4 — <IntRangeList>: "Maximum number of elements: The list must not contain more than 1024 elements."
  • §3.4.1.2 item 4 — <FloatRangeList>: same
  • §3.4.1.3 — <StringRangeList>: same

§3.4.1.1.1 <IntRangeExpr> gives the grammar and exactly one constraint —
"no two ranges in the expression are allowed to overlap" — and no element count.
It also states the form's purpose outright: "The motivating use-case for this
form is providing a succinct way to describe a frame range." Capping the
expansion at 1024 rejects that use case; range: "1-5000" is ordinary
submission traffic for a render farm.

The spec is also explicit elsewhere when it wants to grant implementations
latitude to constrain a count — e.g. for step dependencies: "There is no maximum
defined, though implementations may choose to constrain the number of
dependencies." No such allowance is given for <IntRangeExpr>, and none is
needed: a host service that wants to bound task counts can do so itself, and
CallerLimits::max_task_count exists for exactly that.

Concretely, this is what regressed:

range elements 0.11.0 0.11.1 0.11.2 this PR
1024 accept accept accept accept
1025 accept reject reject accept
10000 accept reject reject accept
DecodeValidationError: 1 validation errors for JobTemplate
steps[0] -> parameterSpace -> taskParameterDefinitions[0] -> INT -> range:
	range expression expands to 10001 elements (max 1024).

Downstream, this pre-empts a host service's own task-count limits. AWS Deadline
Cloud resolves max-tasks-per-step per customer account and can raise it; that
mechanism becomes unreachable above 1024 for a single-parameter range, because
decode now fails before the service's own check runs.

Change

Narrow the cap to the list forms, matching §3.4 as written.

  • Delete _check_range_expr_len and its two call sites: the
    RangeExpressionTaskParameterDefinition._validate_range_len validator (whose
    only job was that check) and the expansion check in
    _validate_int_range_elements.
  • Keep the <IntRangeExpr> grammar validation in
    _validate_int_range_elements — a malformed expression is still rejected.
  • Keep _MAX_TASK_PARAM_RANGE_LEN and validate_task_param_range_list_len
    unchanged; the list-form cap is correct and predates 0.11.1 (it was already
    enforced in 0.11.0 as Field(max_length=1024) on IntRangeList /
    FloatRangeList / StringRangeList).
  • Rewrite the constant's comment to record which forms it governs and why the
    expression form is excluded, so the cap does not get re-broadened.

The Rust side carries the same over-broad cap (max_task_param_range_len
applied to IntRange::Expression in validate_v2023_09/structure.rs and to the
RangeExpr branches of job/create_job/ranges.rs). That lives in the
openjd-model crate this repo consumes from crates.io, so it is fixed
separately; this PR is the pure-Python v0 / v2023_09 half.

Testing

The expression cap had no test coverage, so nothing needed deleting. Added
TestTaskParameterRangeLength pinning the §3.4 boundary from both directions:

  • range expressions at 1024, 1025, 5000 and 1-100000:2 (50000 elements) parse
    at the template layer and the instantiation layer, and the range expands in
    full (asserts len(instantiated.range) == expected_len, so a future silent
    truncation fails the test);
  • list-form ranges of 1025 are still rejected at both layers.

Also added a 1-5000 case to TestRangeExpressionTaskParameterDefinition.

Full suite: 5471 passed, 24 skipped, 3 xfailed.

Signed-off-by: David Leong <leongdl@amazon.com>
@leongdl
leongdl requested a review from a team as a code owner August 14, 2026 18:40
# Do not apply this to an `<IntRangeExpr>` expansion. §3.4.1.1.1 constrains that
# form only by "no two ranges may overlap" and states its purpose is to express
# frame ranges succinctly, so capping the expansion rejects the form's primary
# use case and pre-empts the host service's own task-count limits.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removing the expansion cap leaves an unbounded materialization on the CHUNK[INT] + range-expression path. _MAX_TASK_PARAM_RANGE_LEN was the only thing bounding it.

_step_param_space_iter.py:291-294 expands a CHUNK[INT] range into a real Python list, and then set(...) of it (lines 333, 968, 1049):

if isinstance(parameter.range, list):
    parameter_range: list[int] = [int(v) for v in parameter.range]
else:
    parameter_range = list[int](parameter.range)   # IntRangeExpr -> full expansion

So a template with

- name: Frames
  type: "CHUNK[INT]"
  range: "1-100000000"
  chunks: { defaultTaskCount: 100 }

now parses cleanly (IntRangeExpr.from_str is O(1) — it stores range objects), and any consumer that later builds a StepParameterSpaceIterator allocates ~100M ints in a list plus a set — multiple GB — before it can reject anything. Previously _check_range_expr_len rejected this at parse time. For a library that validates untrusted job templates, that is a memory-exhaustion vector rather than a validation error.

Note the plain INT range-expression path is fine — RangeExpressionIdentifierNode keeps the IntRangeExpr and iterates lazily, and len() is O(1). The problem is specific to the chunk path (and to the range_set/_range_set construction).

If uncapping the expression form is the intended spec reading, the chunk path probably needs to either stop materializing (chunk boundaries over an IntRangeExpr are computable without expanding it, since __getitem__ is O(log n)) or carry its own explicit, documented bound so the failure is a ValidationError and not an OOM.

@seant-aws seant-aws Aug 14, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spec confirms CHUNK[INT] can absolutely use <IntRangeExpr>

RangeExpressionTaskParameterDefinition model once they are resolved.
against the ``<IntRangeExpr>`` grammar; a list-form range is length-capped
(§3.4). The expansion of a range expression is deliberately not capped —
see ``_MAX_TASK_PARAM_RANGE_LEN``.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-implementation parity note. The two comments this PR deletes both asserted that the cap matched openjd-rs (EffectiveLimits.max_task_param_range_len, and the resolve-time checks in create_job/ranges.rs). Dropping the check on the Python side without a corresponding change in openjd-rs means a template with range: "1-5000" now validates here but would still be rejected at create_job, so callers using this library as a pre-submission validator get a late, surprising failure instead of an early one.

Worth confirming the openjd-rs side is being changed in step (or that the earlier comments were simply wrong about what it enforces) — otherwise the two implementations disagree on which templates are valid.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rs changes in progress

@seant-aws seant-aws left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch

@leongdl
leongdl merged commit 3fc3da4 into mainline Aug 14, 2026
31 of 32 checks passed
leongdl added a commit that referenced this pull request Aug 14, 2026
Signed-off-by: David Leong <leongdl@amazon.com>
@leongdl leongdl mentioned this pull request Aug 14, 2026
leongdl added a commit that referenced this pull request Aug 14, 2026
Signed-off-by: David Leong <116610336+leongdl@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants