[docs] Added test guide #5215

Vineet1101 · 2025-04-02T11:12:31Z

This PR adds a comprehensive guide on testing in the P4C repository. The document covers different types of tests, their expected behavior, source file locations, how to run tests, and how to add new tests. Additionally, it includes details on continuous integration and automated test execution.

github-actions · 2025-04-02T11:13:04Z

A preview of this PR is available at:	Preview Link
📂 View the source code here:	View Source Code
🔧 Commit used for deployment:	2da0d7791bfcbb0c44e11050e75d291767dfbc2d

Note: Changes may take a few seconds to appear on GitHub Pages. Please refresh the page if you do not see the updates immediately.

Vineet1101 · 2025-04-02T16:39:00Z

@jafingerhut there is no bmv_errors_output directory in the testdata

jafingerhut · 2025-04-02T17:11:59Z

@jafingerhut there is no bmv_errors_output directory in the testdata

That is true. There is also no directory bmv_errors, either, so why would you expect there to be a directory bmv_errors_output?

jafingerhut · 2025-04-02T17:14:46Z

@jafingerhut there is no bmv_errors_output directory in the testdata

That is true. There is also no directory bmv_errors, either, so why would you expect there to be a directory bmv_errors_output?

If you mean: "Why is there a directory named p4_16_bmv_errors in the testdata directory, but there is no corresponding expected outputs directory named p4_16_bmv_errors_output?" then my answer is "I don't know".

It would be good to find out, and either correct the problem, or document why one is not necessary. It might be a mistake that there is no directory p4_16_bmv_errors_output.

Vineet1101 · 2025-04-02T17:19:43Z

yes I mean that why there is bmv_errors file but no output No problem I will try to find out why there is no output file

@jafingerhut there is no bmv_errors_output directory in the testdata

That is true. There is also no directory bmv_errors, either, so why would you expect there to be a directory bmv_errors_output?

If you mean: "Why is there a directory named p4_16_bmv_errors in the testdata directory, but there is no corresponding expected outputs directory named p4_16_bmv_errors_output?" then my answer is "I don't know".

It would be good to find out, and either correct the problem, or document why one is not necessary. It might be a mistake that there is no directory p4_16_bmv_errors_output.

Vineet1101 · 2025-04-04T02:25:00Z

@jafingerhut can you please review this
also i could not find the reason why there is no output file for p4_16_bmv_errors so I have not included it in the doc for now

fruffy · 2025-04-02T12:29:24Z

docs/AddingTestGuide.md

+
+## Running Tests
+
+- To run all tests: `make check -j3`


Not everyone uses make. I would recommend ctest here. If make is recommended somewhere else, I would update it.

added testing using ctest section and remove cmake one

also running test using make is written in the p4 docs so if you want i can update the docs

Yes, please update the docs. But this section also should be fixed. ctest is how tests are run, not make.

For example, this is how you would run tests using the same command cmake --build build --target check -- -j3

Copilot

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

docs/AddingTestGuide.md:17

The source file location for Negative Tests has been removed without a replacement. Please verify if updated directory paths should be provided to ensure clarity for readers.

- `testdata/p4_16_errors/`

docs/AddingTestGuide.md:28

The Backend-Specific Tests section no longer lists expected source file locations. If these paths have changed, consider updating this section with the current directories to avoid confusion.

- `backends/ebpf/tests/`

Copilot · 2025-04-05T10:18:28Z

docs/AddingTestGuide.md

+Positive tests are standard tests where the compiler should not fail due to compile-time errors. The expected output should match the actual output for the test to pass.The compiler should generate a output file with no errors.
+


There is a missing space after the period in 'pass.The'; please add a space to improve readability.

Suggested change

Positive tests are standard tests where the compiler should not fail due to compile-time errors. The expected output should match the actual output for the test to pass.The compiler should generate a output file with no errors.

Positive tests are standard tests where the compiler should not fail due to compile-time errors. The expected output should match the actual output for the test to pass. The compiler should generate a output file with no errors.

@fruffy can you please review it again?

Copilot

Copilot reviewed 1 out of 1 changed files in this pull request and generated no comments.

Comments suppressed due to low confidence (2)

docs/AddingTestGuide.md:40

[nitpick] The sentence seems a bit awkward; consider rephrasing it for clarity, for example: 'Similarly, for other error output files, please refer to the corresponding directories.'

Similarly for other error output files you can check

docs/AddingTestGuide.md:25

[nitpick] Consider capitalizing 'ubpf' to 'UBPF' for consistency with 'eBPF' to improve clarity.

Some test programs exist in backend-specific directories, such as eBPF, ubpf tests.

jafingerhut · 2025-06-17T17:46:25Z

@fruffy You mentioned in an earlier comment: "Yes, please update the docs. But this section also should be fixed. ctest is how tests are run, not make."

I searched through existing p4c files (latest committed as of 2025-Jun-17), and there are quite a few places that mention commands with make check, make check-sanity, make recheck, etc. sometimes with additional options. See the list below.

I have used make check for 6-7 years for running automated p4c tests. I don't mind changing to use whatever you recommend, but it confuses me when you say "make is not how tests are run" when it is spread throughout the docs, and it seems to still work. I don't claim to know that it works identically to how ctest behaves -- perhaps there are differences.

Are you hoping to deprecate make check and its variants for p4c as a whole, and all its back ends, in favor of using ctest?

./cmake/P4CUtils.cmake:# add rules to make check and recheck for a specific test suite
./backends/tofino/bf-asm/README.md:user@box$ make check
./backends/tofino/bf-asm/README.md:user@box$ make check-sanity
./backends/tofino/bf-asm/README.md:This is similar to `make check` but will only run on .p4 files in the tests
./backends/p4test/run-p4-sample.py:                ' or rerun all tests using "P4TEST_REPLACE=True make check".' % expected
./backends/ebpf/README.md:- `make check-ebpf`: runs the basic ebpf user-space tests
./backends/ebpf/README.md:- `make check-ebpf-bcc`: runs the user-space tests using bcc to compile ebpf
./backends/ebpf/README.md:- `sudo -E make check-ebpf-kernel`: runs the kernel-level tests.
./backends/ebpf/run-ebpf-test.py:            # If the user runs tests with `make -jX check`, the `make -f .../runtime.mk`
./backends/dpdk/run-dpdk-test.py:                ' or rerun all tests using "P4TEST_REPLACE=True make check".' % expected
./docs/README.md:* To run tests execute `make check -j3`
./docs/README.md:* To run a subset of tests execute `make check-PATTERN`.  E.g., `make
./docs/README.md:* To rerun the tests that failed last time run `make recheck`
./docs/README.md:changes are built, and the tests executed via `make check`.  These
./docs/README.md:  * For example, [any P4 file under `testdata/p4_16_samples/`](https://github.com/p4lang/p4c/blob/de2eaa085152abd0690660fbe561eaf5db7b2bf7/backends/p4test/CMakeLists.txt#L63) belongs to the [`p4` test suite](https://github.com/p4lang/p4c/blob/de2eaa085152abd0690660fbe561eaf5db7b2bf7/backends/p4test/CMakeLists.txt#L113) (meaning it will run with `make check-p4`).
./docs/README.md:  * [File ending with `*-bmv2.p4`](https://github.com/p4lang/p4c/blob/de2eaa085152abd0690660fbe561eaf5db7b2bf7/backends/bmv2/CMakeLists.txt#L127) also belongs to the [`bmv2` test suite](https://github.com/p4lang/p4c/blob/de2eaa085152abd0690660fbe561eaf5db7b2bf7/backends/bmv2/CMakeLists.txt#L200) (meaning it will run with `make check-bmv2`).
./docs/README.md:  * If you have many reference outputs to add/update, you could also do `P4TEST_REPLACE=True make check` (or `make check-*`) to update all tests.
./README.md:    make -j4 check
./README.md:We implemented support equivalent to the automake `make check` rules.
./README.md:All tests should be included in `make check` and in addition, we support
./README.md:`make check-*` rules. To enable this support, add the following rules:
./README.md:make check P4C_ARGS="-Xp4c=MY_CUSTOM_FLAG"
./README.md:P4TEST_REPLACE=1 make check

fruffy · 2025-06-17T17:54:19Z

@fruffy You mentioned in an earlier comment: "Yes, please update the docs. But this section also should be fixed. ctest is how tests are run, not make."

I searched through existing p4c files (latest committed as of 2025-Jun-17), and there are quite a few places that mention commands with make check, make check-sanity, make recheck, etc. sometimes with additional options. See the list below.

I have used make check for 6-7 years for running automated p4c tests. I don't mind changing to use whatever you recommend, but it confuses me when you say "make is not how tests are run" when it is spread throughout the docs, and it seems to still work. I don't claim to know that it works identically to how ctest behaves -- perhaps there are differences.

Are you hoping to deprecate make check and its variants for p4c as a whole, and all its back ends, in favor of using ctest?

./cmake/P4CUtils.cmake:# add rules to make check and recheck for a specific test suite
./backends/tofino/bf-asm/README.md:user@box$ make check
./backends/tofino/bf-asm/README.md:user@box$ make check-sanity
./backends/tofino/bf-asm/README.md:This is similar to `make check` but will only run on .p4 files in the tests
./backends/p4test/run-p4-sample.py:                ' or rerun all tests using "P4TEST_REPLACE=True make check".' % expected
./backends/ebpf/README.md:- `make check-ebpf`: runs the basic ebpf user-space tests
./backends/ebpf/README.md:- `make check-ebpf-bcc`: runs the user-space tests using bcc to compile ebpf
./backends/ebpf/README.md:- `sudo -E make check-ebpf-kernel`: runs the kernel-level tests.
./backends/ebpf/run-ebpf-test.py:            # If the user runs tests with `make -jX check`, the `make -f .../runtime.mk`
./backends/dpdk/run-dpdk-test.py:                ' or rerun all tests using "P4TEST_REPLACE=True make check".' % expected
./docs/README.md:* To run tests execute `make check -j3`
./docs/README.md:* To run a subset of tests execute `make check-PATTERN`.  E.g., `make
./docs/README.md:* To rerun the tests that failed last time run `make recheck`
./docs/README.md:changes are built, and the tests executed via `make check`.  These
./docs/README.md:  * For example, [any P4 file under `testdata/p4_16_samples/`](https://github.com/p4lang/p4c/blob/de2eaa085152abd0690660fbe561eaf5db7b2bf7/backends/p4test/CMakeLists.txt#L63) belongs to the [`p4` test suite](https://github.com/p4lang/p4c/blob/de2eaa085152abd0690660fbe561eaf5db7b2bf7/backends/p4test/CMakeLists.txt#L113) (meaning it will run with `make check-p4`).
./docs/README.md:  * [File ending with `*-bmv2.p4`](https://github.com/p4lang/p4c/blob/de2eaa085152abd0690660fbe561eaf5db7b2bf7/backends/bmv2/CMakeLists.txt#L127) also belongs to the [`bmv2` test suite](https://github.com/p4lang/p4c/blob/de2eaa085152abd0690660fbe561eaf5db7b2bf7/backends/bmv2/CMakeLists.txt#L200) (meaning it will run with `make check-bmv2`).
./docs/README.md:  * If you have many reference outputs to add/update, you could also do `P4TEST_REPLACE=True make check` (or `make check-*`) to update all tests.
./README.md:    make -j4 check
./README.md:We implemented support equivalent to the automake `make check` rules.
./README.md:All tests should be included in `make check` and in addition, we support
./README.md:`make check-*` rules. To enable this support, add the following rules:
./README.md:make check P4C_ARGS="-Xp4c=MY_CUSTOM_FLAG"
./README.md:P4TEST_REPLACE=1 make check

Well, I rushed this comment out a bit, but the point is that many are building p4c using ninja nowadays. These instructions won't work with that. We should also fix the mentions of make there.

The recommended way to do this is to use the cmake front end with cmake --build build --target check -- -j3 or similar commands. This will pick make or ninja, depending on the generator.

Dscano · 2025-08-18T19:31:02Z

@Vineet1101 If you have a bit of time, could you take a look at Fabian’s comment so we can get the PR merged? It would be a pity to keep this work pending after all the effort you’ve put into it

ChrisDodd · 2025-08-18T20:18:38Z

docs/AddingTestGuide.md

+    - Tests targeting eBPF should end with -ebpf.p4.
+    - General frontend tests should use descriptive names that reflect the feature being tested.
+3. Generate reference outputs:
+   - For frontend tests: `../backends/p4test/run-p4-sample.py . -f ../testdata/p4_16_samples/some_name.p4`


The easiest way to (re)generate expected outputs for tests is to set P4TEST_REPLACE=1 in the environment before running the test(s), and then run the tests however you normally do (ctest or make check or however you make ninja run them, or even just running .test scripts directly)

ChrisDodd · 2025-08-18T20:27:31Z

docs/AddingTestGuide.md

+#### Source File Location
+- Various locations within `testdata/`
+
+## Running Tests


The way the test suite is designed, is that cmake will create many .test scripts in the build tree (in subdirectoroes named for the target and the test source directory -- eg build/bmv2/testdata/p4_16_samples) that runs that each run a single test, and sets up ctest to run all those scripts. So you can run individual tests by just running a .test script directly from the command line, or you can run ctest, or you can run a build tool to run ctest (eg make check)

Vineet1101 · 2025-08-19T04:33:38Z

@Dscano @fruffy @jafingerhut I will complete this by the end of the day

Signed-off-by: Vineet1101 <Vineetgoel692@gmail.com>

Vineet1101 · 2025-08-23T09:44:57Z

@jafingerhut @fruffy can you please review it I have made changes as per your review

fruffy

@jafingerhut Any other comments?

Signed-off-by: Vineet1101 <Vineetgoel692@gmail.com>

github-actions bot added the documentation Topics related to compiler documentation. label Apr 2, 2025

jafingerhut mentioned this pull request Apr 2, 2025

Skip PSATernaryTest on Ubuntu 22.04 and add Black formatting #5209

Merged

Vineet1101 marked this pull request as ready for review April 4, 2025 02:24

fruffy reviewed Apr 5, 2025

View reviewed changes

fruffy requested a review from Copilot April 5, 2025 10:17

Copilot AI reviewed Apr 5, 2025

View reviewed changes

Vineet1101 requested review from Copilot and fruffy April 5, 2025 18:55

Copilot AI reviewed Apr 5, 2025

View reviewed changes

fruffy requested a review from jafingerhut June 15, 2025 21:50

fruffy mentioned this pull request Aug 18, 2025

Add document to p4c repo explaining to developers a little bit of what they need to know to add new tests p4lang/project-ideas#32

Open

ChrisDodd reviewed Aug 18, 2025

View reviewed changes

Vineet1101 added 4 commits August 23, 2025 14:50

Added initial draft of test guide

bcae8a1

Signed-off-by: Vineet1101 <Vineetgoel692@gmail.com>

added some more detail

c115edb

Signed-off-by: Vineet1101 <Vineetgoel692@gmail.com>

added testing using ctest section and removed cmake section

31edfec

Signed-off-by: Vineet1101 <Vineetgoel692@gmail.com>

updated test guide

124c064

Signed-off-by: Vineet1101 <Vineetgoel692@gmail.com>

Vineet1101 force-pushed the adding-test-guide branch from 72ecc79 to 124c064 Compare August 23, 2025 09:43

Merge branch 'main' into adding-test-guide

a90d33a

fruffy approved these changes Sep 2, 2025

View reviewed changes

Merge branch 'main' into adding-test-guide

2da0d77

fruffy added this pull request to the merge queue Sep 12, 2025

Merged via the queue into p4lang:main with commit d068783 Sep 12, 2025
20 checks passed

jhavrane pushed a commit to jhavrane/p4c that referenced this pull request Dec 19, 2025

[docs] Added test guide (p4lang#5215)

69008d4

Signed-off-by: Vineet1101 <Vineetgoel692@gmail.com>

		Positive tests are standard tests where the compiler should not fail due to compile-time errors. The expected output should match the actual output for the test to pass.The compiler should generate a output file with no errors.

[docs] Added test guide #5215

[docs] Added test guide #5215

Uh oh!

Conversation

Vineet1101 commented Apr 2, 2025

Uh oh!

github-actions bot commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Vineet1101 commented Apr 2, 2025

Uh oh!

jafingerhut commented Apr 2, 2025

Uh oh!

jafingerhut commented Apr 2, 2025

Uh oh!

Vineet1101 commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Vineet1101 commented Apr 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI Apr 5, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

jafingerhut commented Jun 17, 2025 • edited by fruffy Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fruffy commented Jun 17, 2025

Uh oh!

Dscano commented Aug 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Vineet1101 commented Aug 19, 2025

Uh oh!

Vineet1101 commented Aug 23, 2025

Uh oh!

fruffy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

github-actions bot commented Apr 2, 2025 •

edited

Loading

Vineet1101 commented Apr 2, 2025 •

edited

Loading

Vineet1101 commented Apr 4, 2025 •

edited

Loading

jafingerhut commented Jun 17, 2025 •

edited by fruffy

Loading