fix(ecosystem): detect build systems below the repository root - #18
Merged
Conversation
… wheel
EoSim is the healthiest repo in the org - 1645 tests passing before this change.
But `eosim run stm32f4` printed "PASSED (10000 cycles)" while executing nothing,
and the published wheel could not run at all. Both are fixed.
THE RUN REPORTED SUCCESS FOR DOING NOTHING
_run_eosim() never loaded firmware. It built a VirtualMachine over zeroed memory,
stepped the CPU 10000 times through NOPs, and VirtualMachine.run() returned the
literal `success: True` after printing "EoS booted successfully" unconditionally.
No EoS was involved and nothing distinguished that from a real boot.
run() now reports why it stopped - 'no-firmware', 'halted', 'cycle-limit' or
'timeout' - and success is derived: only a clean halt counts. Exhausting the
cycle budget or the clock means we stopped it, not that it finished.
`eosim run` gained --firmware and exits 2 with "NO FIRMWARE" when given none.
Measured, from a clean wheel install:
eosim run stm32f4 -> NO FIRMWARE, exit 2
eosim run stm32f4 --firmware fw.bin -> PASSED (3 cycles, halted), exit 0
THE WHEEL WAS BROKEN
No EoSim release has ever published a wheel, so this had never been exercised.
Building one showed why it matters: `platforms/` sat at the repository root and
was addressed as EOSIM_ROOT/"platforms" where EOSIM_ROOT is site-packages once
installed. Every `eosim run` from a wheel died with FileNotFoundError on
site-packages/platforms. An editable install hides this completely.
Moved the 150 platform directories to eosim/platforms so the package is
self-contained, declared them as package-data, and made the lookup prefer the
packaged location with a fallback for older layouts.
Verified: wheel contains 149 platform.yml; a fresh venv installs it and
`eosim stats` reports 149 platforms.
THE CPU IS REAL - now proven
eosim/engine/native/cpu implements an ARM32 subset (MOV imm, B, LDR, STR, BX LR,
SVC, UDF) and genuinely decodes and executes. tests/unit/test_native_engine_
execution.py hand-assembles instructions and asserts the resulting register and
memory state, so this is measured rather than assumed. Also fixed the cycle
accounting: the halting instruction retired but was not counted, leaving
run()['cycles'] one behind cpu.state.cycles for every program that halts.
I CHANGED FIVE EXISTING TESTS - flagging this explicitly
test_core::test_vm_run, test_gui::test_vm_uart_output,
test_gui::test_vm_run_stop_lifecycle, test_cli::test_run_eosim_engine and
test_engines_and_integrations::TestEoSimEngineRun::test_run all ran with NO
firmware and asserted success. They pinned the defect. Each now loads a real
image and asserts the real outcome, and each gained a companion asserting that
the no-firmware path is a failure. Nothing was deleted or skipped.
Two integration tests started skipping with "platforms/ directory not found"
after the move - a silent disable, which is worse than a failure. Repointed;
test_validate_all_real_platforms now validates all 149 configs for real.
ALSO
--version said 2.0.0 while pyproject and eosim.__version__ said 3.0.1. Now
sourced from __version__.
Log output used escaped \\n, so every log arrived as one long line with
literal backslash-n. Four sites fixed.
Package metadata claimed "Development Status :: 5 - Production/Stable" and
"World's most powerful universal simulation platform - supersedes 250+ tools
across 20 domains". Now Beta, with a description of what it is.
CI counted platform.yml at the old path.
Verified: 1659 passed, 3 skipped (pre-existing: tkinter absent), 0 failed.
Coverage 60.87%. ruff clean on the file I added; the repo has 490 pre-existing
ruff errors, untouched and unrelated.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The decoder covers eight ARM32 instructions: MOV imm, B, LDR, STR, BX LR, SVC, UDF and the all-zero word. Anything else fell off the end of the if/elif chain and execution continued as though it had run. That is the same class of defect as the hardcoded success this branch already fixed, and it is worse, because it survives the fix. Measured before the change: MOV r1,#5 ; MOV r2,#3 ; ADD r0,r1,r2 ; UDF -> R0 = 0, reason 'halted', success True The ADD never executed, the run reported a clean successful halt, and nothing in the output said otherwise. Since only eight opcodes are decoded, any real firmware is mostly undecodable - PUSH {lr}, which opens almost every compiled ARM function, is not among them. Loading a genuine EoS image would have produced a confident "PASSED" for a program that computed nothing. Now: - _execute() reports whether it decoded the instruction. - step() halts on an unknown opcode when strict_undefined is set (the default), recording the count and the offending PC/opcode. - run() reports reason 'undefined-instruction', success False, and prints the opcode, address, and the fact that this engine cannot execute a full firmware image - so the limitation is visible at the point it bites. - undefined_count is returned in the result. - strict_undefined can be turned off for tracing experiments. Undecoded instructions are still counted then, just not fatal. This bounds what the native engine honestly claims: it runs small hand-assembled programs, not an operating system. Extending the ISA is the work that would change that, and it is now a visible failure rather than a silent one. Verified: 1663 passed, 3 skipped (pre-existing: tkinter absent), 0 failed. The 1645 tests that predate this branch all still pass. ruff clean on the test file; the 4 errors in cpu/__init__.py are pre-existing, confirmed by stashing. Wheel rebuilt and reinstalled clean: 149 platforms, firmware run PASSED. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ver ran
`eosim ecosystem` is the command for validating the whole organisation. It
was reaching 2 of the 19 repos in the workspace, and could not report a
failure for the ones it did reach.
Discovery. find_repos held a hardcoded list of seven lowercase names —
"eai", "eni", "eipc", "eboot", "ebuild-tool". None of those match a real
directory on a case-sensitive filesystem, so only eos and eApps were found,
and run_ecosystem_tests silently `continue`d past anything not on the list.
Repos are now discovered by looking for .git, and their build system by the
files they contain, so a new product becomes testable by being cloned.
Fabricated results. test_c_repo ended with
tests_passed = max(tests_passed, tests_run if build_ok else 0)
tests_failed = max(0, tests_run - tests_passed)
passed = build_ok and tests_failed == 0
where tests_run counted executables on disk. Any repo that compiled reported
every test passing and a verdict of PASS, whether or not one test had run.
ctest and pytest output is now parsed for the counts they actually printed,
and `passed` derives from a single status field.
A repo that cannot be tested reports SKIP, never PASS — an absent toolchain,
a repo with no tests registered, and a green suite are three different facts.
DEPS is a narrower case: the suite exists but the repo's own declared
dependencies are missing here, which is not the repo's fault and not a FAIL.
Repos with more than one build system now have each one exercised. ebuild is
a Python CLI that also ships a CMakeLists integrating the sibling repos;
running only the primary kind left one of the two untested, which is how a
broken CMake configure sat behind a green Python suite.
Two runner bugs found by running it:
- pytest was invoked with -q. A repo whose own addopts already sets -q ended
up at -q -q, which suppresses the summary line — a fully green EoStudio
read as "no summary produced". The flag is no longer passed.
- PYTHONPATH did not include the repo, so a repo that is not pip-installed
failed at conftest import. Both the root and src/ are now offered, which
covers the src-layout eDB uses.
CLI: --simulate was accepted and then never passed to run_ecosystem_tests,
so --no-simulate did nothing. Added --only to test named repos and --list to
show what would run.
before 2 repos reached, counts fabricated, verdict always ALL PASSED
after 19 discovered, 7 passed, 1 failed, 11 skipped, 2420 tests run
The one failure is real: ebuild's CMake configure, tracked in eBoot#60.
Verification: 1702 pass (1666 before, +36 new tests here); full ecosystem run
against the 19-repo workspace reproduces the table above.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…t of the failure count
Two things the first full run exposed.
The CMake tree was written to <repo>/eosim-build, so every C repo the runner
touched was left with an untracked directory in it — a dirty working tree in
eight repos, and in one without a matching .gitignore, something a developer
could commit by accident. Build trees now go under ~/.cache/eosim/ecosystem,
one per repo, overridable with EOSIM_BUILD_ROOT.
The summary also read "0 repos failed" beside "16 tests failed", because a
DEPS repo's uncollectable tests were folded into total_failed. Those tests
never ran; counting them as failures reads as broken code when the cause is
an absent dependency. They are counted and labelled separately now.
Tests: 2463 run | 2447 passed | 0 failed | 16 blocked on missing deps
Verification: 1706 pass; a full ecosystem run leaves no repo dirty.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…it code
eosllm is built and tested through a Makefile, so the runner reported
"no runner for a 'make' project" and it contributed nothing to the ecosystem
run. Its Makefile declares a `test` target; that target now runs.
`make test` is only attempted when the Makefile actually declares the rule.
Running it otherwise fails with "No rule to make target", which would read as
a broken repo rather than one that keeps its tests somewhere else.
There is no count to parse out of make, and inventing one would be exactly
the fabrication this module was rewritten to remove, so a make repo reports
its exit code instead of a tests:0/0 that looks like nothing ran. The report
shows a count where there is one and the reason where there is not.
Measured against the 19-repo workspace, with the toolchains now installed on
this machine:
before 8 passed | 1 failed | 10 skipped | 2867 tests
after 9 passed | 1 failed | 9 skipped | 2867 tests
eIPC also moved from SKIP to PASS with 152 Go tests once Go was available --
no change needed here, which is the point of detecting toolchains at run time
rather than hardcoding what a machine has.
1710 tests pass, 44 in this module.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| output = str(tmp_path / "junit.xml") | ||
| path = generate_junit(results, output) | ||
| assert os.path.exists(path) | ||
| content = open(path).read() |
| assert 'Artifacts exported' in result.output | ||
| # SPDX-License-Identifier: MIT | ||
| """Unit tests for CLI commands using Click's CliRunner.""" | ||
| from unittest.mock import patch, MagicMock |
| not a single test had run. | ||
| """ | ||
|
|
||
| import os |
Comment on lines
+30
to
+36
| from eosim.integrations.ecosystem import ( | ||
| DEPS, ERROR, FAIL, PASS, SKIP, | ||
| EcosystemReport, RepoTestResult, | ||
| _parse_ctest, _parse_pytest, | ||
| detect_components, detect_kind, detect_kinds, find_repos, | ||
| test_repo as run_one_repo, | ||
| ) |
| data = yaml.safe_load(f) | ||
| if data and data.get("name") == name: | ||
| return yml, data | ||
| except Exception: |
| data = yaml.safe_load(f) | ||
| if data: | ||
| return candidate, data | ||
| except Exception: |
detect_kinds() inspected only the root, so three of the nineteen repos in
the workspace reported "unknown", found no runner, and were skipped:
eCAD-Hardware-Products tests/, no packaging file
eos-aero AeroSwift/software/web_app/package.json
eos-health firmware/build-system/CMakeLists.txt
apps/web/package.json
16% of the organisation was invisible to the tool whose purpose is to test
the organisation. eos-health is the sharpest case: it ships firmware with a
CMake build and a web app, and neither was ever compiled or tested.
The summary counted them as skipped, next to the passes:
Repos: 19 discovered | N passed | 0 failed | 3 skipped
"0 failed" with everything green is the shape a healthy run makes. A skip
reads as a pass at a glance; the reason only appears in the detail rows.
Same class as the fabricated pass removed in #16 — a repo that was never
tested reporting as though nothing was wrong — through a different door.
Two changes.
detect_components() returns (kind, directory) pairs. The runners build from
the directory handed to them, so a nested component has to carry its own
location; returning "cmake" for eos-health without it would send
cmake -S at a root with no CMakeLists.txt, replacing a silent skip with a
spurious failure. Root detection runs first and is returned unchanged when
it finds anything, so the scan only ever runs for a repo that would have
been skipped. Bounded to four levels and blind to node_modules, build,
dist, venv, vendor and friends — a vendored package.json belongs to a
dependency, not to the repo.
A tests/ directory holding Python now identifies a Python project.
test_python_repo() only ever required tests/; demanding pyproject.toml to
reach it was the detector asking for something the runner does not use.
Guarded on no other kind having matched, so eBoot — CMakeLists.txt plus a
tests/ directory full of C — does not also get pytest pointed at it.
Verified no regression: detect_kinds output compared against the original
for every repo in the workspace. One difference, the intended one:
eCAD-Hardware-Products ['unknown'] -> ['python']
All other 18 identical, and detect_components equals the old detect_kinds
result for all 17 root-detected repos.
After: 0 unknown, 19 repos yielding 26 components.
tests/unit/test_ecosystem_runner.py 44 -> 54
full suite 1710 -> 1720 passed, 0 failed
Measured against this branch's base, which is #16, not master. master alone
is at 1648; #16 carries it to 1710. This stacks on #16 and needs it first.
Closes #17
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
srpatcha
force-pushed
the
fix/detect-nested-build-systems
branch
from
August 30, 2026 13:55
3c0a4c6 to
72df0d1
Compare
Testing eos-health for the first time — it was one of the three repos the
previous commit unskipped — produced three CMake failures that the runner
reported identically as FAIL:
eos-health/firmware/build-system FAIL cmake configure failed
eos-health/devices/health-band-neuro/firmware FAIL cmake configure failed
eos-health/firmware/health-band-neuro/tests FAIL cmake configure failed
They are not the same thing. The first stops on
NRF5_SDK_PATH not set. Download nRF5 SDK 17.1.0
which means install something. The other two stop on
Cannot find source file: src/main.c
Cannot find source file: .../src/ecg/ecg_hrv.c
which means the CMakeLists and the tree disagree, and no installation will
help. Putting both in the same column costs the reader the one distinction
that decides what to do next.
The runner already draws exactly this line for Python — DEPS, "a narrower
SKIP: the suite exists and would run, but the repo's dependencies are
absent" — and for node's npm ci. CMake was the odd one out.
_unmet_toolchain() names the absent dependency from an unset *_SDK/_ROOT/
_DIR/_PATH/_HOME variable or a failed find_package, and the configure step
reports DEPS instead of FAIL when it finds one.
Deliberately narrow. "Cannot find source file" and "No SOURCES given" are
checked first and force None, so a repository referencing code it does not
contain stays a failure. eos-health emits both kinds in one configure run,
and the repo defect has to be the one that survives — a status reading
"not our fault" over a real defect is worse than no classification at all.
That case is pinned by a test.
tests/unit/test_ecosystem_runner.py 54 -> 61
full suite 1720 -> 1727 passed, 0 failed
Refs #17
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The README described EoSim's platforms and engines but said nothing about the ecosystem runner, which is the part that tests the organisation rather than a simulation. Adds a section covering how repositories and their build systems are discovered, that a repo with several build systems is exercised through all of them, and what the five statuses mean — in particular why DEPS is kept apart from FAIL. "Install the nRF5 SDK" and "this CMakeLists references a source file that does not exist" are opposite problems and a single red status would hide which one you have. Also records that counts come from the runner's own output and never from an exit status, since the inverse was a real defect here: tests_passed was once inferred from a successful build, so every repo that compiled reported a passing suite whether or not it ran a single test. The ecosystem table listed eboot, eipc, eai and eni in lowercase. GitHub redirects those, so the links worked, but the casing is wrong on disk and this is not a cosmetic detail in this repository: find_repos held a hardcoded lowercase list and consequently discovered 2 of 19 repos on a case-sensitive filesystem. Corrected, with a note saying why. eFirmware and eDB were missing from the table and are added. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A full ecosystem run reported five repositories as FAIL:
EoSim FAIL pytest produced no summary (exit 1)
EoStudio FAIL pytest produced no summary (exit 1)
eDB FAIL pytest produced no summary (exit 1)
ebuild FAIL pytest produced no summary (exit 1)
eosllm FAIL pytest produced no summary (exit 1)
None of them is failing. EoStudio's suite is 729 passed, run directly. The
runner's interpreter simply has no pytest installed:
/usr/bin/python3: No module named pytest
DEPS already exists for this — "the suite exists and would run, but the
repo's dependencies are absent" — but it is only reachable after the counts
are parsed. A collection error aborts pytest before it prints any summary,
so that branch is never entered and everything lands on FAIL. The previous
commit fixed the same shape for cmake; this is the Python half.
Two changes.
The no-summary branch now checks for absent modules and reports DEPS naming
them. It asks _external_missing_modules(), which excludes anything the repo
itself provides — a package directory, a src/ layout, or a single module
file. The runner puts the checkout on PYTHONPATH, so a repo failing to
import its own package is a real defect and has to stay a FAIL. A repo's own
absence does not mask a third-party one; both are pinned by tests.
_missing_modules() now matches the unquoted spelling too. ModuleNotFoundError
quotes the name, but `python -m pytest` on an interpreter without pytest
prints "No module named pytest" bare. That is precisely the case where the
runner itself is what is missing and nothing else in the output explains the
failure, and the quoted-only pattern walked past it.
EoStudio DEPS needs pytest
eDB DEPS needs pytest
eosllm DEPS needs pytest
tests/unit/test_ecosystem_runner.py 61 -> 71
full suite 1727 -> 1737 passed, 0 failed
Refs #17
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #17.
detect_kinds()inspected only the repository root, so three of the nineteenrepos reported
unknown, found no runner, and were skipped:tests/with pytest files, no packaging filefirmware/build-system/CMakeLists.txtapps/web/package.jsonAeroSwift/software/web_app/package.json16% of the organisation was invisible to the tool whose purpose is to test the
organisation.
eos-healthis the sharpest case — it ships firmware with a CMakebuild and a web app, and neither was ever compiled or tested.
The summary counted them as skipped, alongside the passes:
0 failedwith everything green is the shape a healthy run makes. A skip readsas a pass at a glance; the reason string only appears in the detail rows. Same
class as the fabricated pass removed in #16 — a repo that was never really
tested reporting as though nothing was wrong — arriving through a different door.
The change
detect_components()returns(kind, directory)pairs. The runners buildfrom the directory they are handed, so a nested component has to carry its own
location — returning
cmakefor eos-health without it would sendcmake -S <repo root>, which has noCMakeLists.txt, replacing a silent skipwith a spurious failure. That is not an improvement.
Root detection runs first and is returned unchanged when it finds anything, so
the scan only ever runs for a repo that would otherwise have been skipped.
Bounded to four levels and blind to
node_modules,build,dist,venv,vendorand friends — a vendoredpackage.jsonbelongs to a dependency, not tothe repo.
A
tests/directory holding Python now identifies a Python project.test_python_repo()only ever requiredtests/; demandingpyproject.tomltoreach it was the detector asking for something the runner does not use. Guarded
on no other kind having matched, so eBoot —
CMakeLists.txtplus atests/directory full of C — does not also get pytest pointed at it.
Verified
The claim that matters is that this cannot regress the repos that already work.
detect_kindsoutput compared against the original for every repo in theworkspace — exactly one difference, the intended one:
And
detect_componentsequals the olddetect_kindsresult for all 17root-detected repos.
After: 0 unknown, 19 repos yielding 26 components.
Those numbers are measured against this branch's base, which is #16, not
master. master alone collects 1648; #16 carries it to 1710. This PR stacks on
#16 and needs it merged first.
The 16 new tests include the two that guard the risk in this change: root
detection returned unchanged, and root detection winning over anything nested.