Skip to content
Merged
2 changes: 1 addition & 1 deletion rust-bindings/src/expr/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -26,5 +26,5 @@ pub(crate) use path_mapping::PyPathMappingRule;
pub(crate) use profile::{PyExprExtension, PyExprProfile, PyExprRevision, PyHostContext};
pub(crate) use range_expr::{PyIntRange, PyRangeExpr};
pub(crate) use symbol_table::{
_reconstruct_serialized_symtab, PySerializedSymbolTable, PySymbolTable,
_reconstruct_serialized_symtab, build_symbol_table, PySerializedSymbolTable, PySymbolTable,

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #285 — EXPR / WRAP_ACTIONS: key call stacks

Repo: OpenJobDescription/openjd-model-for-python · branch python-rs
What the PR does: teaches the pure-Python (v0) model to handle the RFC
0005–0008 expression extensions by delegating {{ }} parsing, type-checking,
and evaluation to the Rust openjd-expr engine over PyO3 bindings.

TL;DR

Before this PR the pure-Python model only understood the legacy
Name.Dot.Name interpolation grammar. This PR makes it a thin front-end over
the Rust expression engine whenever a template declares the EXPR extension,
and adds the parameter types and wrap-action rules that ride on top of it.

  • EXPR expressions — when EXPR is declared, {{ }} parsing/eval routes to
    a new Rust-backed ExprNode (arithmetic, functions, lists, paths, let,
    comprehensions) instead of the legacy name parser. Non-EXPR templates are
    byte-for-byte unchanged, and the Rust surface is imported lazily so the
    non-EXPR path never loads it.
  • Type-aware validation — job-parameter types are mapped to EXPR types and
    fed to a new ParsedExpression.typecheck() that catches real type errors at
    decode time without false-rejecting runtime-dependent expressions (no more
    error-string sniffing).
  • 8 new job-parameter types (RFC 0007) — BOOL, RANGE_EXPR, and the
    LIST[*] family — with case-insensitive type names, item/length constraints,
    and an EXPR-extension gate, all sharing one base class.
  • Native values end-to-endParameterValue.value/JobParameter.value
    widen from str to Any so lists/bools survive create_job intact; a new
    SymbolTable.expr_types field carries the type tags so the Rust builder
    coerces correctly (and is preserved across copy/union).
  • Spec-correct string coercion — format-string output now uses the engine's
    RFC 0005 coercion via evaluate_to_str() ({{ true }}"true", null →
    "", quoted list items) instead of Python's str().
  • WRAP_ACTIONS (RFC 0008)onWrapEnvEnter/onWrapTaskRun/onWrapEnvExit on
    EnvironmentActions, gated on WRAP_ACTIONS+EXPR, all-or-nothing, with
    per-hook Wrapped* variable scoping and a job-wide single-wrap-layer rule.
  • New Rust bindingsbuild_symbol_table, ParsedExpression.typecheck,
    job_parameter_type_expr_spec (single-sources the OpenJD→EXPR type mapping).

Risk shape: almost all behavior change is gated behind the EXPR /
WRAP_ACTIONS extension declarations, so templates that don't opt in are
unaffected. The sharp edges are at the Python↔Rust boundary (the three flows
that cross it) and the str → Any value widening.


Two flows form the spine of the change: (1) decode-time validation of an
expression's symbol references, and (2) runtime resolution of an expression
into the string substituted into a format string. ──BOUNDARY──▶ marks a call
crossing from Python into compiled Rust; ◀── marks the return.


Flow 1 — Decode-time: validate an EXPR {{ }} expression's symbols

Entry: decode_job_template(...) → pydantic validation → the variable-reference
prevalidator.

Flow: prevalidate_model_template_variable_references(values, context)
  1. _variable_reference_validation.py:137   prevalidate_model_template_variable_references
  2.   :166                                   expr_enabled = "EXPR" in declared & in context.extensions
  3.   :173                                   context._symbol_types = _build_job_parameter_symbol_types(values)
  4.     └─ :433                              _build_job_parameter_symbol_types
  5.        └─ _expr_support.py               expr_type_for_openjd_type("LIST[INT]")
  6.           └──BOUNDARY──▶                 _openjd_rs job_parameter_type_expr_spec   (model/types.rs)
  7.                                          uppercase-parse + OpenJD→EXPR map   ◀── "list[int]" | None
  8.   :460                                   _check_format_string(value, scope, symbols, context)
  9.     :484                                 expr.validate_symbol_refs(symbols, symbol_types=ctx._symbol_types)
 10.        └─ _expression.py                 InterpolationExpression → self._expresion_tree.validate_symbol_refs
 11.           └─ _nodes.py:149              ExprNode.validate_symbol_refs(symbols, symbol_types)
 12.              :150                        accessed = self._parsed.accessed_symbols   (free dotted refs, from Rust)
 13.              :167                        for name: longest_defined_prefix(name, symbols) → prefix_types
 14.              :173    ┌── fully typed ──  validate_typed_expression(self._parsed, typed_symbols=prefix_types)
 15.              │       └─ _expr_support.py  values = {dotted: ExprValue.unresolved(ExprType(t))}
 16.              │          └──BOUNDARY──▶    _openjd_rs ParsedExpression.typecheck   (parsed_expression.rs)
 17.              │                            eval vs unresolved placeholders, DISCARD result,
 18.              │                            raise ONLY on a genuine type error   ◀──
 19.              └── else (some prefix untyped) ─ name-only fallback:
 20.                    :182                   missing = accessed - symbols
 21.                    :184                   if missing: raise _missing_symbol_error(name, symbols)

Key insight: step 16's typecheck() discards the result, so a "well-typed
but unresolved at runtime" expression passes cleanly — the model no longer has
to sniff Rust error-message strings to tell a real type error from a
runtime-dependent one. (The let-binding path reaches the same
ExprNode.validate_symbol_refs from _variable_reference_validation.py:606,
building one ExprNode per binding.)


Flow 2 — Runtime: resolve a {{ }} expression to its substituted string

Entry: a caller invokes FormatString.resolve(symtab=...) during job
instantiation / argument resolution.

Flow: FormatString.resolve(symtab, path_format)  —  {{ }} → substituted string
  1. _format_string.py:79             FormatString.resolve(symtab, path_format)
  2.   :115                           element.expression.evaluate_to_str(symtab, path_format)
  3.     └─ _expression.py            InterpolationExpression → self._expresion_tree.evaluate_to_str
  4.        └─ _nodes.py:234          ExprNode.evaluate_to_str(symtab, path_format)
  5.           :241 / :208            value = self._evaluate_raw(symtab, path_format)
  6.              └─ :221             values = symtab_to_expr_values(symtab, types=symtab.expr_types or None)
  7.                 └─ _expr_support.py   map OpenJD types → EXPR specs (Flow 1, steps 5–7)
  8.                    └──BOUNDARY──▶ _openjd_rs build_symbol_table(flat, expr_types, path_format)
  9.                                   coerce "10":INT → int, nest "Param.Frame"   ◀── typed SymbolTable
 10.              :225 └──BOUNDARY──▶  _openjd_rs ParsedExpression.evaluate(values, profile, path_format)  ◀── ExprValue
 11.           :242                    if value.is_null: return ""        # null → empty string (RFC 0005)
 12.           :244                    return str(value)                  # ExprValue.__str__ = engine spec coercion
 13.   :127                           resolved_list.append(resolved_value)   # NOTE: str() wrapper removed here

Key insight: step 12's str(ExprValue) is the fix for the
{{ true }}"True" Python-repr bug — it uses the engine's RFC 0005 coercion
(true/false, double-quoted list items, preserved Decimal trailing zeros),
not Python's str(). Step 9 is where the expr_types metadata earns its keep:
if that map is ever dropped on a SymbolTable copy/union, coercion silently
reverts to inference and produces a wrong-typed value — which is exactly why the
PR promotes expr_types to a real field that is copied in __init__ and
union. ExprNode.evaluate (_nodes.py:231) instead calls .item() to hand
back the native Python value for callers that want the typed result.


Flow 3 — Decode-time: a new typed job parameter (BOOL / LIST[*] / RANGE_EXPR)

The PR adds eight EXPR-extension job-parameter types. They share one base class
and a normalize-then-discriminate decode path. Entry: decode_job_template(...)
on a template whose parameterDefinitions contains, say, {"type": "list[int]"}.

Flow: JobTemplate.parameterDefinitions validation
  1. _model.py:3809  @field_validator("parameterDefinitions", mode="before")
  2.    └─ :3427      _normalize_parameter_type_case(value, info)
  3.       :3434      if "EXPR" in context.extensions:  item["type"] = item["type"].upper()
  4.                  # "list[int]" → "LIST[INT]"  (RFC 0007 case-insensitive type names)
  5.    │  (pydantic now resolves the discriminated union on the upper-cased "type")
  6.    └─ Literal[JobParameterType.LIST_INT] ─▶ JobListIntParameterDefinition
  7.       └─ class _JobListParameterDefinitionBase (_model.py:3477)
  8.          :3487   @field_validator("type")  _validate_type_gate → _expr_param_gate
  9.                  if context & "EXPR" not in extensions: raise "requires the EXPR extension"
 10.          :3519   @model_validator(after) _validate_default
 11.             └─ :3512  _check_list_value(self.default)
 12.                       _check_list_length(name, v, minLength, maxLength)
 13.                       for item: self._check_item(item)   # subclass hook
 14.                         └─ JobListIntParameterDefinition._check_item → _check_int_item
 15.                            └─ _check_numeric_item_bounds(...)   # shared int/float bounds (deduped this PR)

Key insight: every concrete list type is just a type literal + an item
constraint + a one-line _check_item override on the shared base — the length
check, EXPR gate, create-job metadata, and template-variable defs all live once
on _JobListParameterDefinitionBase. RANGE_EXPR (_model.py:3644) is the odd
one out: it validates its default string through the existing IntRangeExpr
grammar rather than a list check. Two-stage decode (normalize before,
discriminate after) is why a lowercase list[int] resolves to the right
class instead of failing the union.


Flow 4 — Instantiation: create_job carries native values for the new types

Legacy scalars (STRING/INT/FLOAT/PATH) are stringified through preprocessing;
the new EXPR types must keep their native Python value (a real list/bool) so
the typed symbol-table builder (Flow 2, step 8) can coerce them. Entry:
create_job(job_template, job_parameter_values).

Flow: create_job(template, values)
  1. _create_job.py:293   create_job
  2.   :64                _collect_defaults_2023_09
  3.     :79              is_legacy_scalar = param.type.name in _LEGACY_SCALAR_TYPE_NAMES
  4.     :82  ┌─ default ─ if not is_legacy_scalar:  ParameterValue(value=param.default)   # native, no str()
  5.     :123 └─ provided ─ if not is_legacy_scalar:  ParameterValue(value=value)          # native, no str()
  6.                       (legacy branch still does str(value) + PATH-relative join below)
  7.   :139                _check_2023_09(defs, values)
  8.     :155              check_constraints = getattr(param, "_check_constraints", None)
  9.                       if check_constraints: check_constraints(param_value.value)      # list/range checked here
 10.   :349                expr_types = {}; for each non-legacy param:
 11.     :357              expr_types["Param.<n>"] = expr_types["RawParam.<n>"] = param.type.value
 12.     :359              symtab.expr_types.update(expr_types)   # ← feeds Flow 2 step 8 coercion

Key insight: this is the producer side of the expr_types contract that
Flow 2 consumes. The widened ParameterValue.value: Any (was str) is what lets
a [1, 2, 3] survive instantiation intact; the review confirmed no downstream
consumer string-ops that value, and the type tag travels alongside it via
symtab.expr_types so the Rust builder coerces correctly.

Merge wrinkle — _merge_job_parameter.py

When the same parameter is defined in multiple templates, only the four legacy
scalar types take the cross-merge path (merge_job_parameter_definitions_for_one
:191, gated on _LEGACY_CONSTRAINT_TYPES). EXPR types skip it and return the
last-defined definition via model_copy (:241) — which bypasses validators, so
the PR re-runs getattr(base, "_check_constraints", None) (:235) on the carried
default to avoid letting a constraint-violating default slip through.


Flow 5 — Decode-time: WRAP_ACTIONS structural validation (RFC 0008)

Two independent checks fire when WRAP_ACTIONS is declared: per-EnvironmentActions
shape/scope, and a job-wide single-wrap-layer rule.

Flow: EnvironmentActions + JobTemplate WRAP_ACTIONS validation
  A. _model.py:516   EnvironmentActions._requires_oneof  (mode=before)
       any_wrap = any of (onWrapEnvEnter/onWrapTaskRun/onWrapEnvExit) set
       if any_wrap & context present:
         require "WRAP_ACTIONS" in extensions; require "EXPR" (hard prereq)
         require ALL THREE wrap hooks (all-or-nothing)
       # context-gated: skipped at create_job re-validation (no context)
  B. _model.py:565   EnvironmentActions._validate_wrapped_variable_scope (mode=after)
       for field in (onEnter, onExit, *wrap hooks):
         referenced = _action_referenced_namespaces(action)   # :398, scans command/args/timeout FormatStrings
            └─ expr.accessed_symbols → namespace = symbol.split(".")[0]   # WrappedAction/WrappedEnv/WrappedStep
         reject namespaces not allowed in that hook (WrappedStep only in onWrapTaskRun, etc.)
  C. _model.py:3941  JobTemplate._validate_single_wrap_layer (mode=after)
       job_env_wraps = count of jobEnvironments with a wrap hook (_env_defines_wrap_hook :373)
       for each step: if job_env_wraps + step's stepEnvironment wraps > 1: reject
       # "at most one wrap layer reachable per session stack"

Key insight: the three checks live at different altitudes on purpose —
extension/all-or-nothing gating per EnvironmentActions (before), per-hook
variable scoping per EnvironmentActions (after), and the cross-cutting
single-layer count at JobTemplate (after, where it can see jobEnvironments +
all steps). The scope check (B) deliberately inspects the timeout FormatString
too, not just command/args, because timeout is a format string under
FEATURE_BUNDLE_1.


Reference — type changes that ripple through the flows

Change File Why it matters
ParameterValue.value: str → Any _types.py lets BOOL/LIST values ride through preprocessing natively (Flow 4); a stringified list can't be re-parsed into a typed list
JobParameter.value: str → Any v2023_09/_model.py same, on the instantiated-job side
SymbolTable.expr_types: dict[str,str] (new field) _symbol_table.py the type-tag map; copied in __init__/union so coercion (Flow 2 step 8) never silently reverts to inference
ExpressionInfo.resolved_value: Optional[Union[Real,str]] → Optional[str] _format_string.py resolve() now appends the value directly (str() wrapper dropped); evaluate_to_str guarantees a string
Node.allows_nonscalar + evaluate_to_str() (new) _nodes.py lets EXPR nodes return list/bool/null while the legacy FullNameNode stays str/Real-only
validate_symbol_refs(..., symbol_types=...) (new kwarg) _nodes.py / _expression.py threads the type map into the typed-validation fork (Flow 1 step 11)
DefinesTemplateVariables.expr_inject (new) _types.py EXPR-only symbol injection (e.g. Step.Name) applied by the prevalidator, which has the parsing context
New Rust bindings: build_symbol_table, ParsedExpression.typecheck, job_parameter_type_expr_spec rust-bindings/ the three boundary calls the flows above cross into

};
31 changes: 31 additions & 0 deletions rust-bindings/src/expr/parsed_expression.rs
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,37 @@ impl PyParsedExpression {
Ok(PyExprValue { inner: value })
}

/// Type-check the expression against a symbol table of (typically
/// unresolved) typed placeholders, without extracting a concrete value.
///
/// Succeeds when the expression is well-typed for the given symbol types —
/// including when the result is an unresolved value that merely depends on
/// a runtime symbol — and raises only on a genuine type/evaluation error.
/// Unlike :meth:`evaluate`, it discards the result, so it never raises the
/// "cannot extract value from unresolved" boundary error; callers no longer
/// need to sniff the error message to tell a real type error from a
/// runtime-dependent one.
#[pyo3(signature = (*, values=None, profile=None))]
fn typecheck(
&self,
values: Option<&Bound<'_, pyo3::PyAny>>,
profile: Option<&PyExprProfile>,
) -> PyResult<()> {
let symtab;
let symtab_refs: Vec<&SymbolTable> = if let Some(v) = values {
symtab = extract_symtab(v)?;
vec![&symtab]
} else {
vec![]
};
let lib = profile_for_call(profile);
self.inner
.with_library(&lib)
.evaluate(&symtab_refs)
.map_err(expr_err_to_py)?;
Ok(())
}

/// Evaluate the expression and return an :class:`EvalResult` with the
/// resulting value alongside the per-call resource-usage metrics
/// (``peak_memory`` in bytes, ``operation_count``).
Expand Down
82 changes: 81 additions & 1 deletion rust-bindings/src/expr/symbol_table.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,17 @@
// SPDX-License-Identifier: Apache-2.0

use pyo3::prelude::*;
use pyo3::types::PyDict;
use pyo3::types::{PyDict, PyString};
#[cfg(feature = "stub-gen")]
use pyo3_stub_gen::derive::*;

use openjd_expr::path_mapping::PathFormat;
use openjd_expr::symbol_table::SymbolTable;
use openjd_expr::types::ExprType;
use openjd_expr::value::ExprValue;

use crate::expr::expr_value::{py_to_expr_value, PyExprValue};
use crate::expr::path_format::PyPathFormat;

#[cfg_attr(feature = "stub-gen", gen_stub_pyclass(module = "openjd._openjd_rs"))]
#[pyclass(module = "openjd.expr", name = "SymbolTable", from_py_object)]
Expand Down Expand Up @@ -324,3 +328,79 @@ pub(crate) fn _reconstruct_serialized_symtab(json: &str) -> PyResult<PySerialize
})?;
Ok(PySerializedSymbolTable { inner })
}

// ── Typed symbol-table builder ─────────────────────────────────────
//
// Bridges the pure-Python (v0) model's flat dotted-key symbol table into
// a typed `SymbolTable` the engine can evaluate against. This replaces the
// former Python `symtab_to_expr_values`/`_to_expr_value` coercion so the
// string→typed-value coercion lives next to the engine (PR #285 review,
// C3/C5). The OpenJD-type → EXPR-type-spec mapping stays in Python; this
// function takes the resolved EXPR type spec strings (e.g. "int",
// "list[int]", "path") so the engine owns only the coercion + nesting.

/// Coerce a single Python value to an `ExprValue`, optionally toward a known
/// target `ExprType`. String values are coerced via `from_str_coerce` (so a
/// stored ``"10"`` of type INT becomes a real integer); other native values
/// are built then coerced. With no target the value's native type is
/// inferred. Mirrors the former Python ``_expr_support._to_expr_value``.
fn coerce_symbol_value(
value: &Bound<'_, pyo3::PyAny>,
target: Option<&ExprType>,
pf: PathFormat,
) -> PyResult<ExprValue> {
let Some(target) = target else {
// No confident type — let the engine infer from the native value.
return py_to_expr_value(value);
};
if let Ok(s) = value.cast::<PyString>() {
return ExprValue::from_str_coerce(&s.to_cow()?, target, pf)
.map_err(pyo3::exceptions::PyValueError::new_err);
}
py_to_expr_value(value)?
.coerce(target, pf)
.map_err(pyo3::exceptions::PyValueError::new_err)
}

/// Build a typed :class:`SymbolTable` from a flat dotted-key value map and an
/// optional per-key EXPR type-spec map, coercing string values to their
/// declared type and nesting dotted keys (``"Param.Frame"``) into subtables.
///
/// ``values`` maps dotted symbol names to their (typically string) values, as
/// the v0 model stores them. ``types`` maps the same dotted names to EXPR
/// type spec strings (``"int"``, ``"list[int]"``, ``"path"``, …); names absent
/// from ``types`` are inferred from the value. ``path_format`` controls how
/// PATH-typed values are interpreted (defaults to the host OS).
#[cfg_attr(
feature = "stub-gen",
gen_stub_pyfunction(module = "openjd._openjd_rs")
)]
#[pyfunction]
#[pyo3(signature = (values, types=None, *, path_format=None))]
pub(crate) fn build_symbol_table(
values: &Bound<'_, PyDict>,
types: Option<&Bound<'_, PyDict>>,
path_format: Option<PyPathFormat>,
) -> PyResult<PySymbolTable> {
let pf = path_format
.map(PathFormat::from)
.unwrap_or_else(PathFormat::host);
let mut st = SymbolTable::new();
for (key, value) in values.iter() {
let dotted: String = key.extract()?;
let target: Option<ExprType> = match types {
Some(t) => match t.get_item(dotted.as_str())? {
Some(spec_obj) => {
let spec: String = spec_obj.extract()?;
Some(ExprType::parse(&spec).map_err(pyo3::exceptions::PyValueError::new_err)?)
}
None => None,
},
None => None,
};
let ev = coerce_symbol_value(&value, target.as_ref(), pf)?;
st.set(&dotted, ev)
.map_err(|e| pyo3::exceptions::PyValueError::new_err(e.to_string()))?;
}
Ok(PySymbolTable { inner: st })
}
2 changes: 2 additions & 0 deletions rust-bindings/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,7 @@ fn openjd_rs(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(evaluate_expression, m)?)?;
m.add_function(wrap_pyfunction!(parse_expression, m)?)?;
m.add_function(wrap_pyfunction!(escape_format_string, m)?)?;
m.add_function(wrap_pyfunction!(build_symbol_table, m)?)?;
m.add_function(wrap_pyfunction!(_reconstruct_expr_value, m)?)?;
m.add_function(wrap_pyfunction!(_reconstruct_serialized_symtab, m)?)?;

Expand Down Expand Up @@ -231,6 +232,7 @@ fn openjd_rs(m: &Bound<'_, PyModule>) -> PyResult<()> {
m.add_function(wrap_pyfunction!(validate_attribute_capability_name, m)?)?;
m.add_function(wrap_pyfunction!(standard_amount_capability_names, m)?)?;
m.add_function(wrap_pyfunction!(standard_attribute_capability_names, m)?)?;
m.add_function(wrap_pyfunction!(job_parameter_type_expr_spec, m)?)?;
m.add_function(wrap_pyfunction!(standard_attribute_capabilities, m)?)?;

register_renamed_exception(
Expand Down
4 changes: 2 additions & 2 deletions rust-bindings/src/model/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -72,8 +72,8 @@ pub(crate) use template_types::{
PyStepScript as PyTemplateStepScript, PyStepTemplate,
};
pub(crate) use types::{
PyDocumentType, PyJobParameterType, PyJobParameterValue, PyTaskParameterType,
PyTaskParameterValue, PyTemplateSpecificationVersion,
job_parameter_type_expr_spec, PyDocumentType, PyJobParameterType, PyJobParameterValue,
PyTaskParameterType, PyTaskParameterValue, PyTemplateSpecificationVersion,
};
pub(crate) use user_interfaces::{
PyBoolUserInterface, PyFileFilter, PyFloatUserInterface, PyHiddenOnlyUserInterface,
Expand Down
16 changes: 16 additions & 0 deletions rust-bindings/src/model/types.rs
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,22 @@ impl PyJobParameterType {
}
}

/// Map an OpenJD job-parameter type spec name (e.g. ``"INT"``, ``"LIST[INT]"``,
/// ``"RANGE_EXPR"``; case-insensitive) to its EXPR type spec string
/// (``"int"``, ``"list[int]"``, ``"range_expr"``), or ``None`` when the name is
/// not a recognized job-parameter type. Single-sources both the
/// (case-insensitive) type-name parsing and the OpenJD-type -> EXPR-type
/// mapping in the Rust ``openjd-model`` crate so the Python model does not
/// hand-maintain a parallel table.
#[cfg_attr(
feature = "stub-gen",
gen_stub_pyfunction(module = "openjd._openjd_rs")
)]
#[pyfunction]
pub(crate) fn job_parameter_type_expr_spec(type_name: &str) -> Option<String> {
JobParameterType::from_spec_str(type_name.trim()).map(|t| t.expr_type().to_string())
}

impl From<PyJobParameterType> for JobParameterType {
fn from(v: PyJobParameterType) -> Self {
match v {
Expand Down
15 changes: 15 additions & 0 deletions src/openjd/_openjd_rs.pyi
Original file line number Diff line number Diff line change
Expand Up @@ -1780,6 +1780,12 @@ class ParsedExpression:
@property
def expr(self) -> builtins.str: ...
def __repr__(self) -> builtins.str: ...
def typecheck(
self,
*,
values: typing.Optional[typing.Any] = None,
profile: typing.Optional[ExprProfile] = None,
) -> None: ...
def evaluate(
self,
*,
Expand Down Expand Up @@ -3567,6 +3573,15 @@ def deserialize_step(step_dict: dict) -> Step:
"""

def escape_format_string(value: builtins.str) -> builtins.str: ...
def job_parameter_type_expr_spec(
type_name: builtins.str,
) -> typing.Optional[builtins.str]: ...
def build_symbol_table(
values: builtins.dict,
types: typing.Optional[builtins.dict] = None,
*,
path_format: typing.Optional[PathFormat] = None,
) -> SymbolTable: ...
def evaluate_expression(
expr: builtins.str,
*,
Expand Down
51 changes: 44 additions & 7 deletions src/openjd/model/_create_job.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,12 @@

__all__ = ("preprocess_job_parameters",)

# The original scalar job-parameter type names whose values are carried as
# strings through preprocessing. EXPR-extension types (BOOL, RANGE_EXPR, and
# the LIST[*] variants) are carried natively instead so the typed EXPR symbol
# table can coerce them.
_LEGACY_SCALAR_TYPE_NAMES = frozenset({"STRING", "INT", "FLOAT", "PATH"})


# =======================================================================
# ================ Preprocessing Job Parameters =========================
Expand Down Expand Up @@ -70,8 +76,18 @@ def _collect_defaults_2023_09(
return_value: JobParameterValues = dict[str, ParameterValue]()
# Collect defaults
for param in job_parameter_definitions:
is_legacy_scalar = param.type.name in _LEGACY_SCALAR_TYPE_NAMES
if param.name not in job_parameter_values:
if param.default is not None:
if not is_legacy_scalar:
# EXPR types (BOOL / RANGE_EXPR / LIST[*]): carry the native
# default through so the typed symbol-table builder can
# coerce it. The PATH-relative-default handling below only
# applies to the scalar PATH type.
return_value[param.name] = ParameterValue(
type=ParameterValueType(param.type), value=param.default
)
continue
default = str(param.default)
# Make PATH defaults relative to job_template_dir, and
# enforce the `allow_job_template_dir_walk_up` parameter request.
Expand Down Expand Up @@ -104,6 +120,12 @@ def _collect_defaults_2023_09(
else:
# Check the parameter against the constraints
value = job_parameter_values[param.name]
if not is_legacy_scalar:
# EXPR types: carry the provided native value through.
return_value[param.name] = ParameterValue(
type=ParameterValueType(param.type), value=value
)
continue
# Join any provided relative PATH parameter value with the current_working_directory (except the empty value "")
if param.type.name == "PATH" and value != "" and not Path(value).is_absolute():
value = str(current_working_dir / value)
Expand All @@ -123,8 +145,16 @@ def _check_2023_09(
for param in job_parameter_definitions:
if param.name in job_parameter_values:
param_value = job_parameter_values[param.name]
# The EXPR-extension LIST[*]/RANGE_EXPR definitions don't implement
# _check_constraints (BOOL and the original scalars do). Their
# template defaults are validated at decode time, and their values
# are type-checked when coerced into the typed EXPR symbol table, so
# skip the create-time constraint check when it isn't available.
check_constraints = getattr(param, "_check_constraints", None)
if check_constraints is None:
continue
try:
param._check_constraints(param_value.value)
check_constraints(param_value.value)
except ValueError as err:
errors.append(str(err))

Expand Down Expand Up @@ -312,14 +342,21 @@ def create_job(
if job_template.specificationVersion == TemplateSpecificationVersion.JOBTEMPLATE_v2023_09:
from .v2023_09 import ValueReferenceConstants as ValueReferenceConstants_2023_09

# EXPR-extension typed params (BOOL / RANGE_EXPR / LIST[*]) carry native
# values; record their OpenJD type so the typed symbol-table builder
# coerces them to the right ExprType during expression evaluation. The
# original scalar types keep their existing string-based handling.
expr_types: dict[str, str] = {}
for name, param in all_job_parameter_values.items():
prefix = ValueReferenceConstants_2023_09.JOB_PARAMETER_PREFIX.value
raw_prefix = ValueReferenceConstants_2023_09.JOB_PARAMETER_RAWPREFIX.value
if param.type != "PATH":
symtab[f"{ValueReferenceConstants_2023_09.JOB_PARAMETER_PREFIX.value}.{name}"] = (
all_job_parameter_values[name].value
)
symtab[f"{ValueReferenceConstants_2023_09.JOB_PARAMETER_RAWPREFIX.value}.{name}"] = (
all_job_parameter_values[name].value
)
symtab[f"{prefix}.{name}"] = all_job_parameter_values[name].value
symtab[f"{raw_prefix}.{name}"] = all_job_parameter_values[name].value
if param.type.name not in _LEGACY_SCALAR_TYPE_NAMES:
expr_types[f"{prefix}.{name}"] = param.type.value
expr_types[f"{raw_prefix}.{name}"] = param.type.value
symtab.expr_types.update(expr_types)
else:
raise NotImplementedError(
f"Spec version {job_template.specificationVersion} not implemented."
Expand Down
Loading
Loading