feat(tests): Unify e2e test runner and enable parallel ISO 15118 execution by Pietfried · Pull Request #1940 · EVerest/EVerest

Pietfried · 2026-03-11T10:02:14Z

Describe your changes

Replace per-suite shell scripts with a single tests/run-tests.sh entry point covering all test suites (core, framework, async, ocpp). CI workflows are updated to invoke the new run-tests.sh directly.

ISO 15118 tests are grouped via xdist_group("ISO15118") to prevent conflicts when running without network isolation. When veth pairs are available (created by the new setup-network-isolation.sh), the NetworkIsolationPlugin strips the group markers and assigns each xdist worker a dedicated interface, enabling fully parallel execution. The NetworkIsolationStrategy auto-injects the assigned interface into EVerest configs for ISO 15118 modules.

pytest configuration is consolidated into tests/pytest.ini, which is always selected via --config-file so that both the unified runner and direct pytest invocations use the same rootdir and option definitions.

Issue ticket number and link

Checklist before requesting a review

I have performed a self-review of my code
I have made corresponding changes to the documentation
I read the contribution documentation and made sure that my changes meet its requirements

Pietfried · 2026-03-11T15:51:33Z

EVerest/everest-ci#150 will allow container to execute ip link ... to create the veth pairs as part of the test orchestration.

.ci/build-kit/scripts/create_ocpp_tests_image.sh

.ci/build-kit/scripts/create_integration_image.sh

mlitre

Except for the small nits it looks good

mlitre · 2026-03-23T17:25:31Z

tests/run-tests.sh

+        # CTRL+C at the sudo prompt cancels it and falls back to sequential execution.
+        if "$SETUP_SCRIPT" setup "$WORKERS" 2>/dev/null; then
+            ISOLATION_FLAG="--network-isolation"
+        elif sudo "$SETUP_SCRIPT" setup "$WORKERS"; then


We could probably set the -n or else it might hang here right?

Its actually intentional for local developer use. The CI uses the non-sudo path that runs first. Its a bit fragile, which is why I added the comment above.

mlitre · 2026-03-23T17:27:15Z

applications/utils/everest-testing/src/everest/testing/core_utils/network_isolation/plugin.py

+        if not self._active:
+            return
+
+        stripped_count = 0


Do you need this, you don't do anything with it

Good catch, removed it

mlitre · 2026-03-23T17:28:21Z

tests/ocpp_tests/conftest.py

+    except ValueError:
+        pass


Maybe we should log something here or else we never know if there was an error

added a log

mlitre · 2026-03-23T17:29:37Z

tests/core_tests/smoke_tests.py

        "gcp": [Requirement("grid_connection_point", "external_limits")],
    }
 )
+@pytest.mark.xdist_group(name="ISO15118")


Why is this needed here? Isn't this basic charging

true, removed it

mlitre · 2026-03-23T17:29:53Z

tests/core_tests/smoke_tests.py

        "gcp": [Requirement("grid_connection_point", "external_limits")],
    }
 )
+@pytest.mark.xdist_group(name="ISO15118")


true, removed it

.github/workflows/job_manifest-tests.yml

mlitre

The OCPP test most likely fails in the CI due to the strict timing so either we can put it in a serial run or relax the timing

mlitre · 2026-03-24T21:47:12Z

applications/utils/everest-testing/src/everest/testing/core_utils/network_isolation/strategy.py

+from everest.testing.core_utils import EverestConfigAdjustmentStrategy
+
+# Module types that use the `device` config field for ISO 15118 communication
+ISO15118_MODULE_TYPES = frozenset({"EvseV2G", "Evse15118D20", "PyEvJosev"})


@Pietfried I think your missing the network isolation for the IsoMux module, so if you just do the following it should fix your issue.

Suggested change

ISO15118_MODULE_TYPES = frozenset({"EvseV2G", "Evse15118D20", "PyEvJosev"})

ISO15118_MODULE_TYPES = frozenset({"EvseV2G", "Evse15118D20", "IsoMux", "PyEvJosev"})

I thought so too, but its unfortunately not that easy. IsoMux connects to the -2 and -20 modules via lo on ::1 so the tests would conflict. To address this we need to make the proxy device IsoMux connects to not hardcoded but configurable. @SebaLukas fyi
@mlitre @andistorm I am putting this back to draft and will potentially add a PR for IsoMux which can then be incorporated in this PR.

…ution Replace per-suite shell scripts with a single tests/run-tests.sh entry point covering all test suites (core, framework, async, ocpp). CI workflows are updated to invoke run-tests.sh directly. ISO 15118 tests are grouped via xdist_group("ISO15118") to prevent conflicts when running without network isolation. When veth pairs are available (created by the new setup-network-isolation.sh), the NetworkIsolationPlugin strips the group markers and assigns each xdist worker a dedicated interface, enabling fully parallel execution. The NetworkIsolationStrategy auto-injects the assigned interface into EVerest configs for ISO 15118 modules. pytest configuration is consolidated into tests/pytest.ini, which is always selected via --config-file so that both the unified runner and direct pytest invocations use the same rootdir and option definitions. Signed-off-by: Piet Gömpel <pietgoempel@gmail.com> Co-authored-by: Piet Gömpel <37657534+Pietfried@users.noreply.github.com> Signed-off-by: Piet Gömpel <37657534+Pietfried@users.noreply.github.com>

Signed-off-by: Piet Gömpel <pietgoempel@gmail.com>

Pietfried mentioned this pull request Mar 11, 2026

feat(build-env-base): Add iproute2 package. This will allow EVerest E2E tests to dynamically create veth pairs EVerest/everest-ci#150

Merged

Pietfried commented Mar 11, 2026

View reviewed changes

.ci/build-kit/scripts/create_ocpp_tests_image.sh Outdated Show resolved Hide resolved

Pietfried commented Mar 11, 2026

View reviewed changes

.ci/build-kit/scripts/create_integration_image.sh Outdated Show resolved Hide resolved

Pietfried force-pushed the tests/e2e/iso15118-parallelization branch 2 times, most recently from b797f9d to a181f64 Compare March 16, 2026 15:33

Pietfried marked this pull request as ready for review March 16, 2026 15:33

Pietfried requested review from SebaLukas, andistorm, cburandt, corneliusclaussen, djchhp, florinmihut, hikinggrass, james-ctc and mlitre as code owners March 16, 2026 15:33

Pietfried marked this pull request as draft March 16, 2026 15:36

Pietfried marked this pull request as ready for review March 17, 2026 08:37

Pietfried assigned andistorm and mlitre Mar 17, 2026

mlitre reviewed Mar 23, 2026

View reviewed changes

Pietfried force-pushed the tests/e2e/iso15118-parallelization branch from a181f64 to 32131b2 Compare March 23, 2026 17:43

Pietfried requested a review from mlitre March 23, 2026 17:44

andistorm reviewed Mar 24, 2026

View reviewed changes

.github/workflows/job_manifest-tests.yml Outdated Show resolved Hide resolved

Pietfried force-pushed the tests/e2e/iso15118-parallelization branch 2 times, most recently from 0424d03 to 1ff0188 Compare March 24, 2026 15:44

mlitre reviewed Mar 24, 2026

View reviewed changes

Pietfried marked this pull request as draft March 25, 2026 07:27

Pietfried mentioned this pull request Mar 25, 2026

feat(IsoMux): Make proxy device configurable #2004

Merged

3 tasks

Pietfried added 2 commits March 26, 2026 10:48

Adress requested changes

dd100b7

Signed-off-by: Piet Gömpel <pietgoempel@gmail.com>

Incorporate proxy device configuration for IsoMux

59d5d0b

Signed-off-by: Piet Gömpel <pietgoempel@gmail.com>

Pietfried force-pushed the tests/e2e/iso15118-parallelization branch from 1ff0188 to 59d5d0b Compare March 26, 2026 09:48

Pietfried marked this pull request as ready for review March 26, 2026 13:01

mlitre self-requested a review March 26, 2026 14:54

mlitre approved these changes Mar 26, 2026

View reviewed changes

Pietfried enabled auto-merge March 26, 2026 15:06

Pietfried requested a review from andistorm March 26, 2026 15:06

	ISO15118_MODULE_TYPES = frozenset({"EvseV2G", "Evse15118D20", "PyEvJosev"})
	ISO15118_MODULE_TYPES = frozenset({"EvseV2G", "Evse15118D20", "IsoMux", "PyEvJosev"})

Conversation

Pietfried commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes

Issue ticket number and link

Checklist before requesting a review

Uh oh!

Pietfried commented Mar 11, 2026

Uh oh!

Uh oh!

Uh oh!

mlitre left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mlitre left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Pietfried commented Mar 11, 2026 •

edited

Loading