Skip to content

test(suite): document deliberate non-goals in lua53 suite#216

Merged
davydog187 merged 3 commits into
mainfrom
triage/sandbox-refusals
May 9, 2026
Merged

test(suite): document deliberate non-goals in lua53 suite#216
davydog187 merged 3 commits into
mainfrom
triage/sandbox-refusals

Conversation

@davydog187
Copy link
Copy Markdown
Contributor

Triage cluster — sandbox-refusal suite files

Plan: .agents/plans/A20-triage-sandbox-refusals.md

Goal

Decide, per file, whether each of main.lua, verybig.lua, files.lua,
attrib.lua is a permanent non-goal (skip with explanatory tag) or whether
the suite runner should be configured to stub os.execute / os.tmpname /
os.getenv so the file completes. Produce per-file follow-up plans or skip
annotations.

Decision: all four are skip-permanently. Each exercises capabilities
that conflict with this library's role as a sandboxed embedded Lua VM.

Per-file decisions

File Failure Decision Reason
main.lua os.execute() is sandboxed skip-permanently Tests the standalone Lua interpreter binary: shells out, writes Lua programs to temp files, invokes lua as a subprocess. We are an embedded VM with no shell-out.
files.lua os.getenv(_) is sandboxed skip-permanently Tests filesystem I/O end-to-end. io.* is a stub by design.
attrib.lua module 'io' not found skip-permanently Tests require semantics that depend on writing files to disk and dynamically loading them.
verybig.lua os.tmpname() is sandboxed skip-permanently Tests RK opcodes / >64K constants via a os.tmpname() + io.output() + dofile() harness. The VM behaviour is interesting; the harness is not. A future plan could stub these for the suite runner only — not in scope here.

Success criteria

  • Each of the four files has a documented decision. All four are
    skip-permanently; reasons live in test/lua53_suite_test.exs
    @deferred_permanent map.
  • test/lua53_suite_test.exs gains a comment alongside the
    @tag :skip for each of the four, and this is reflected in
    CHANGELOG.md (Unreleased) and ROADMAP.md (Deferred section).
  • No fix-now follow-up plans needed. (See Discoveries — verybig.lua
    could justify a future stub-for-suite plan, but it is recorded as a
    possibility, not a commitment.)
  • No regression. mix test reports 1585 tests, 0 failures, 31 skipped
    (same as pre-flight). Suite test reports 29 tests, 0 failures, 24
    skipped (5 ready / 4 deferred-permanent / 20 missing-feature).

Changes

 .agents/plans/A20-triage-sandbox-refusals.md |  2 +-
 CHANGELOG.md                                 |  9 +++++
 ROADMAP.md                                   | 22 ++++++++++-
 test/lua53_suite_test.exs                    | 57 +++++++++++++++++++++++++---
 4 files changed, 82 insertions(+), 8 deletions(-)

Key change in test/lua53_suite_test.exs: split the single @skipped_tests
list into @skipped_tests (missing features, future plans welcome) and
@deferred_permanent (a %{file => reason} map for deliberate non-goals).
The deferred-permanent tests get their own describe block with the reason
embedded in the test name and a @tag :deferred_permanent for filtering.

Discoveries

  • The plan's premise that all four files fail with os.execute() /
    os.tmpname() / os.getenv(_) was correct for three of them.
    attrib.lua is a different sandbox policy: it dies on require"io"
    because there is no io global to cache. The cluster framing still
    holds — all four exercise capabilities that conflict with our sandbox
    role — but the failure mechanism for attrib.lua is package/loader-
    driven, not os.*-driven.
  • verybig.lua is the most interesting of the four. The first ~140 lines
    test genuinely useful VM behaviour (RK opcodes, >64K constants), but
    they live inside a generated Lua program written to a temp file and
    executed with dofile(). A future plan could stub these primitives in
    the suite runner only to unlock that coverage. Recorded as a
    possibility, not a commitment.
  • README has no "supported coverage" section. Adding one would duplicate
    what's already in ROADMAP.md's "Deferred" list. Updated ROADMAP and
    CHANGELOG instead.

Verification

mix format
mix compile --warnings-as-errors
mix test
52 doctests, 51 properties, 1585 tests, 0 failures, 31 skipped

Out of scope (intentional)

  • Implementing os.execute etc. for general consumption.
  • Changing the public sandbox API.
  • Stubbing os.tmpname/io.output/dofile for the suite runner —
    recorded as a possible future plan, not shipped here.
  • Suite files outside this cluster.

davydog187 added 3 commits May 8, 2026 19:03
Split the lua53 skipped-tests list into 'missing features' (still
addressable) and 'deferred permanent' (deliberate non-goals). The four
files in the latter group — main.lua, files.lua, attrib.lua,
verybig.lua — exercise shell-out, file I/O, or filesystem-backed
require semantics that conflict with this library's role as a
sandboxed embedded Lua VM. Per-file rationale now lives inline in
test/lua53_suite_test.exs and in ROADMAP's Deferred section.

No behavioural change. Suite count unchanged (5/29 ready, 24 skipped).
Full mix test: 1585 tests, 0 failures, 31 skipped.

Plan: A20
@davydog187 davydog187 merged commit ad04d4e into main May 9, 2026
4 checks passed
@davydog187 davydog187 deleted the triage/sandbox-refusals branch May 9, 2026 12:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant