IPv6 Compliance Summary: 83 failures of IPv6 Core Conformance Tests #28502
Comments
LiveFreeAndRoam
added
the
bug 🐛
|
In the past week, I've managed to upgrade our embedded system from systemd v250 to v253. I have rerun a subset of the tests, ("Base" tests), against systemd v253. The results have improved a little: "Base" Test Failures:
There are still too many failures to make it practical to raise individual issues. I've attached a zip file that lists the failing "Base" tests for v250 and v253. |
|
Most of this is from Linux kernel implementation choices. |
|
On July 13, Matt (the OP) posted on the mailing list:
If his tests are correct, it doesn't seem to be a kernel issue. But, of course, it would be interesting if he could provide more details about the test. |
|
Correct, this is not a kernel issue. Evidently, networkd is part of the IPv6 protocol exchange. It intercepts the IPv6 RAs and disables the kernel's sysctl accept_ra, so the kernel is removed from the RA conversation. Unfortunately, networkd is not handling the RAs in a compliant fashion. I reported 7 issues (#28421, #28434, #28435, #28437, #28438, #28439, #28449) before realising there were so many more and it was not practical to keep reporting these one by one.
IPv6 Forum created the test specifications for IPv6 conformance and interoperability testing. More can be read about it at IPv6 Ready Logo Program. The test suite has been around since the early days of IPv6 development. It is used by IPv6 Protocol developers to ensure they have delivered a compliant implementation of the IPv6 protocol. Over the years, I have worked with Linux kernel developers, who also use the test suite, to address a small number of issues that arise from time to time. Product developers use the test suite for certification, as various customers (understandably) demand the IPv6 protocol is compliant. Now that networkd has its hands in the IPv6 protocol, then it too must be developed with IPv6 compliance as a goal. This test suite is the tool that will ensure that goal is met. |
|
@LiveFreeAndRoam thanks for taking the time to do this, Im afraid if you want each issue fixed and not lost in the sea of bugreports you will have to file one issue per problem. otherwise things might never be corrected. |
|
I don't think it makes much sense to report those issues (and fix them) one by one without integrating that test suite into the CI one way or another. That part of networkd is undertested and tends to break every time it changes so with no new tests all those one-off fixes are likely to make things even worse.
The problem is that the commercial test suite can't be used upstream. The TAHI project doesn't seem to be actively maintained any more but other than that it comes with a custom licence containing what seems to be a non-commercial clause so it can't be integrated into the upstream CI easily either. |
|
A little history/context that might be useful. Hideaki Yoshifuji @yoshfuji worked on the project (USAGI Project) which added IPv6 conformance tests to the Linux kernel, using the TAHI tests. Unfortunately, he doesn't seems active anymore, with his last activity on Github on February 2020 [1]. On his publication with Keio University, he he emphasized that "to maintain the stack stable, we developed an automatic testing system, which greatly helps us saving our time." It would be interesting to know how the USAGI project circumvented those custom license problems, considering that the full work was mainlined on the Linux kernel. |
|
Looking at
I think it should be fine to run the test suite downstream somewhere without including the code of the test anywhere or changing it. The systemd test suites are kept and maintained upstream in the repository though and I think that clause would prevent that. |
|
That licence was in
and that's not very reassuring. Either way it would be interesting to know how the USAGI project circumvented those custom license problems. It seems to me that the safest way would be to use the spec to implement that stuff from scratch but it's a huge undertaking. |
|
@evverx I failed to run |
|
@ssahani it isn't up to me :-) Personally I don't think it makes sense to manually file a lot of issues, verify all the patches and catch all sorts of regressions in the process manually but it's just my opinion. |
Yes I hope @LiveFreeAndRoam will be able to attach. |
|
Before I forget #28969 seems to be somewhat related to this issue. |
|
It talks about RFC6334 . I think only RA's M and O bit. |
|
I am willing to help. I feel we can get into a rhythm after we get through a few of these cases. Is there a way we can setup a meet to coordinate and plan an approach? @ssahani, I sent you an email. |
|
Hi @ssahani, I have sent an email to the three of you. In the attached zip file, there are two tar files:
These tar files contain my logs and pcapng files for each of the test cases in the IPv6 Logo certification test suite. In the cases where networkd fails, the legacy version can be examined for the correct behaviour. Let me know if you need more info or otherwise how I can help address these. |
|
@LiveFreeAndRoam So, individual issues has been already fixed and closed, except for #31624. Could you re-run whole test suite again with the current git main? And if there is no new issue, please close this. Unfortunately, still I have no idea about #31624. Still investigating. |
|
Agreed. With just one test that needs to be resolved, I am closing this issue. For the record... From the IPv6 Core Host Conformance Test Suite (PDF) there was initially 105 failing tests out of a total 416 tests. As an aside, the reason this issue lists just 83-failing tests was because the results of other tests were still pending. Thanks to the great work by @yuwata, we now have just 1 failing test. The full list of tests that failed is below:
|
|
I wonder if it would be possible to run this test suite and networkd under ASan/UBSan just in case? There were already heap-use-after-frees like #31485 and crashes introduced in the process so it would be great to run the whole test suite just in case. |
|
I've read a little about ASan and UBSan. Do you have a particular set of configuration options you'd like to use when compiling and running under UBSan and ASan? Periodically, I run the full test suite. It takes about 7 hours to run. So, knowing UBSan config options ahead of time will be prudent. |
|
I think it should be documented in https://github.com/systemd/systemd/blob/main/docs/TESTING_WITH_SANITIZERS.md. (The "clang" paragraph targets the systemd test suite though where it's necessary to pass The unit file should be adjusted too to be compatible with ASan/UBSan. The upstream test suite comments out because they interfere with the sanitizers. |
@LiveFreeAndRoam Wow! Thank you! Your help is much appreciated. |
|
A headsup... I have started testing with UBSan and ASan and now quite a few tests are failing, currently 56. The I'm using: I'm launching systemd-networkd directly via VSCode's debugger. To capture a few things in case it matters... launch.json file{
// Use IntelliSense to learn about possible attributes.
// Hover to view descriptions of existing attributes.
// For more information, visit: https://go.microsoft.com/fwlink/?linkid=830387
"version": "0.2.0",
"configurations": [
{
"name": "(gdb) Launch systemd-networkd",
"type": "cppdbg",
"request": "launch",
"program": "${workspaceFolder}/build/systemd-networkd",
"envFile": "${workspaceFolder}/.env",
"args": [],
"stopAtEntry": false,
"cwd": "${fileDirname}",
"environment": [],
"externalConsole": false,
"MIMode": "gdb",
"miDebuggerPath": "/home/matt/bin/sudo-gdb",
"setupCommands": [
{
"description": "Enable pretty-printing for gdb",
"text": "-enable-pretty-printing",
"ignoreFailures": true
},
{
"description": "Set Disassembly Flavor to Intel",
"text": "-gdb-set disassembly-flavor intel",
"ignoreFailures": true
}
],
"preLaunchTask": "Make systemd",
}
]
}.env fileSYSTEMD_LOG_LEVEL=debug
ASAN_OPTIONS=strict_string_checks=1:detect_stack_use_after_return=1:check_initialization_order=1:strict_init_order=1
UBSAN_OPTIONS=print_stacktrace=1:print_summary=1:halt_on_error=1diff of meson.build to incorporate sanitize feature$ git diff meson.build
diff --git a/meson.build b/meson.build
index e2de148095..423e0f41f8 100644
--- a/meson.build
+++ b/meson.build
@@ -410,6 +410,7 @@ possible_common_link_flags = [
]
c_args = get_option('c_args')
+c_args += '-Db_sanitize=address,undefined'
# Our json library does not support -ffinite-math-only, which is enabled by -Ofast or -ffast-math.
if (('-Ofast' in c_args or '-ffast-math' in c_args or '-ffinite-math-only' in c_args) and '-fno-finite-math-only' not in c_args)systemd-networkd.service changes for sanitizeOnce this test run is complete, I'll disable the UBSan and ASan and rerun the tests. |
Debuggers are usually incompatible with the leak sanitizer so it usually complains when networkd is stopped: ==3857==LeakSanitizer has encountered a fatal error.
==3857==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==3857==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)Even when networkd is launched directly without debuggers the leak sanitizer is usually unhappy because networkd drops privileges. In terms of making it work it would be better to run it using the unit file or |
I'm not sure it works. Assuming it's built with |
|
Something like --- a/meson.build
+++ b/meson.build
@@ -9,6 +9,7 @@ project('systemd', 'c',
'sysconfdir=/etc',
'localstatedir=/var',
'warning_level=2',
+ 'b_sanitize=address,undefined',
],
meson_version : '>= 0.60.0',
)(without |
|
Thank you @evverx. As you suspected, my Perhaps your diff can make it into the @yuwata, can you confirm the commmit that I should be testing. Yesterday's testing with I retested the Redirect failure using your latest fork and the test passed. For my reference, the previous successful test run used: |
|
@LiveFreeAndRoam Please test with the main branch of systemd/system. I have not changed client side recently, and no pending client side change is waiting in network-next branch. The simple test case for Redirect message in our test suite is fine, IIRC. |
I'm not sure it should go to |
systemd version the issue has been seen with
250
Used distribution
Embedded Linux - Debian-based
Linux kernel version used
5.15.71-rt51+gc36e774d0d9a
CPU architectures issue was seen on
aarch64
Component
systemd-networkd
Expected behaviour you didn't see
Pass IPv6 Core Conformance tests.
Unexpected behaviour you saw
Encountered 83 IPv6 Core Conformance failures.
Please find attached a zip file containing:
tc_legacy_failures_summary.txt: 6 failing casestc_networkd_failures_summary.txt: 83 failing casesThe
legacyfile was captured when our host was configured using Debian's legacy/etc/network/interfacesfile andnetworkdwas disabled and stopped. Thenetworkdfile was captured after configuringnetworkdto manage the interface.The
txtfiles just list the failing test case numbers. To decode those, you will need to look them up in the test specification.failures.zip
Steps to reproduce the problem
Run one of the conformance test suites that is referenced on IPv6 Ready Logo Page.
References:
Additional program output to the terminal or log subsystem illustrating the issue
No response
The text was updated successfully, but these errors were encountered: