Refactor - #2
Conversation
* Moved the `specs/simple_specs/tests.json` to
`specs/liquid_ruby/specs.yml`
* They aren't equivalent but are both generated from `Shopify/liquid`
* Built a *Source* abstraction to load in specs from YAML files
* The plan is to extend to a custom txt file format for hand crafted
specs.
* Implmeneted a TestGenerator to define each spec as it's own test
* Implmeneted a default LiquidRuby adapter for running the tests
* This is where other Liquid implementations (e.g. LiquidWasm can hook
into the specs).
Create ruby.yml
|
👋 It seems that this PR is adding a GitHub Action workflow that looks a lot like a CI step. For private repositories, the only supported way of running Action workflows is via the self-hosted runners, and not all use cases for GitHub Actions are supported. Actions should be used primarily for GitHub automation under the new paradigm. You can learn more about the supported use cases in our GitHub Actions documentation. Please migrate your workflow to use shopify-build instead. If you have any questions about how to setup a If you think this workflow has been mistakenly labeled as a CI step, please let us know by commenting your reasons below. cc @Shopify/code-scale |
|
👋 It seems that this PR is adding, or editing, a GitHub Action workflow that is not configured to use our self-hosted GitHub Action runners. For private repositories, the only supported way of running Action workflows is via the self-hosted runners. You can learn more about the supported use cases in our GitHub Actions documentation. To make sure that your workflow is using these runners you need to add the runs-on: self-hostedAll of our runners are Linux based and hosted on x64 machines, with Docker support enabled, so cc @Shopify/code-scale |
Driven by analysis of /tmp/liquid-spec-results.jsonl (6.2M run lines, 8523 specs): find specs that consistently fail at low complexity without actionable guidance. Hints + complexity (#1, #2): - Add specific spec-level hints to 19 consistently-failing specs: the to_liquid-in-conditions/lookups cluster (VariableTest#test_*_calls_to_liquid_value_*, 19 specs at c=220, ~89% fail), the nil-comparison quirk (StatementsTest#test_zero_lq_or_equal_one_involving_nil, c=140, 89% fail), and the nested {% liquid %} spec (LiquidTagTest#test_nested_liquid_tag, c=130, 50% fail). - Raise ForTagTest#test_iterate_with_each_when_no_limit from c=70 to c=200: it uses instantiate:LoaderDrop (drop iteration) and was misplaced in the near-beginner band. Spec-quality gate (#3): - New test_specs_through_220_have_spec_level_hints: requires a spec-LEVEL hint (spec.hint, not suite/file effective_hint) for c<=220, with a frozen per-spec baseline (test/spec_hint_baseline.txt). Bidirectional: fails on new no-hint specs (regressions) AND stale baseline entries (hints added or specs removed), so the grandfather list shrinks instead of accreting dead weight. Threshold stays at 220 per the stated policy; generated/bulk specs are exempted exactly, not by file. - scripts/generate_spec_hint_baseline.rb regenerates the baseline. Filter-matrix hints (#4): - docs/filter_matrix_quirks.md: category-level guide for the 716 c=160 generated filter specs (date rendering, float formatting, type coercion, nil/empty, drops). - tasks/helpers.rb#format_and_write_specs accepts metadata: and writes the _metadata wrapper, so rake generate:standard_filters preserves the file-level hint instead of erasing it. standard_filters.yml metadata hint now references the doc and lists the five failure categories. Runner output (user request): - Default plain-mode output is now a single summary line then the lowest-complexity failures: "Complexity bar cleared: NN, NNNN passes, NNN failures. Next best specs to work on:" + N failure rows (default 5, --max-failures). Each failure row shows [c=N] complexity. Preamble, per-suite progress, and skipped-suite output are verbose-only (-v); --json is unchanged. --add-specs output also gated to verbose. - Updated test/runner_diagnostics_test.rb: new test_plain_output_starts_with_... asserts stdout starts with the summary and omits preamble; prioritized test asserts no "Prioritized Specs" before the header.
1 participant
specs/simple_specs/tests.jsontospecs/liquid_ruby/specs.ymlShopify/liquidspecs.
into the specs).