Assembler Testing

[slontis]

Is there a plan to add CI and tests for all the assembler calls?
We have had multiple issues related to untested assembler recently so this seems like a good idea.

---

Intel SDE

Getting Intel SDE into github actions CI.. Will need to check that the license is ok to do this. As it is an emulator we would only want to run this on a smaller test set. There are some marketplace github actions for SDE that have a MIT license
(e.g. https://github.com/petarpetrovt/setup-sde)

(It looks like you can run the emulator using)
sde [-option] -- exe_name [args]

---

TESTS

@davidben has suggested copying boringssl
(See #21522 (comment))

This is perlasm code written for multiple platforms and c++ that would need conversion to c.

This requires understanding what all the code is doing (as well as registers, unwinding and trampoline functions).
It would also require multiple reviewers to also understand it.

It is hooking into low level calls to do the checking at the deprecated low level API level. (Not sure if these will eventually be removed) and then calling the global assembler ABI which it wishes to check… So they kind of need to be internal tests. The internal tests we currently have use the test framework which is perl based, so this may not fit nicely into the current system. So it might then need its own test system?

[slontis]

@zkWildfire Are you still looking into this?

[zkWildfire]

Sorry about that. It took work about a month to get everything sorted out on the legal side of things. I sent in the CLA for my company and myself to the email listed on OpenSSL's site for CLAs around two weeks ago now but haven't heard back. As such, I'm not sure where I stand from a legal standpoint on whether I can actively contribute to OpenSSL. I did take note of the updates in the last few weeks on the bug and would like to help wrap this up if possible.

[slontis]

@paulidale or @mattcaswell - could you look into this CLA submission?

[t8m]

I was investigating the use of IntelSDE and at least at that time it seemed to me prohibitively slow to run all the OpenSSL tests under it.

[levitte]

How slow are we talking? Could it run as a daily check on branches? Weekly?

[zkWildfire]

Running code under the SDE is extremely slow. The code in the MCVE I created for the AVX512 XMM registers bug (#21522) runs almost instantly when not under the SDE but takes a handful of seconds under the SDE. If the SDE were to be integrated into CI/CD, it would have to run a very specific subset of tests.

That said, it may not be necessary to compile under the SDE, depending on what you have in mind. For the bug that I addressed, the issue is in semi-handwritten assembly, e.g. the assembly emitted by perl scripts. That assembly code is present regardless of whether the CPU used for building supports AVX-512. If your concern is testing the handwritten assembly, then you can compile on any platform without using emulation, then take the compiled binaries and run those under the SDE. This is the process that I followed when confirming the bug.

[davidben]

There is definitely no need to compile under SDE, whether it's assembly or C. The compiler is perfectly capable of targeting whatever CPU it wants without running on that CPU.

[zkWildfire]

Oh also, I forgot to mention - it's not sufficient to run the tests under the SDE. The only way a test would fail when run under the SDE is if it happens to make use of the XMM registers in a way that makes the test fail when the registers are improperly zeroed out. The test would need to both be run under the SDE and run code like what I have in my MCVE where it looks at the register values before and after specific method invocations to ensure that the method isn't breaking the ABI conventions.

[davidben]

Right. See the second half of the initial post for that part. :-) BoringSSL has been running tests like that for just under 5 years now. They are indeed incredibly valuable and I hope OpenSSL adopts some kind of testing framework of this nature.

[davidben]

#22384 is an example of something that'd probably have been caught faster with SDE in CI.

[davidben]

It is hooking into low level calls to do the checking at the deprecated low level API level. (Not sure if these will eventually be removed) and then calling the global assembler ABI which it wishes to check… So they kind of need to be internal tests.

It's not a high vs low level API thing, but that you ideally want to test the asm functions directly. Testing up the stack will work, but the more C functions in the call stack, the more likely you miss bugs due to what the compiler happened to emit for the C calls. Also calling conventions get complicated with too many arguments, or parameters that aren't word-sized. It's a lot easier to say that you don't allow such type signatures in internal asm functions than to constrain the public API.