Add library for compile time instantiation of elliptic curves #3979

randombit · 2024-04-06T10:24:57Z

It slices, it dices.

Everything is constant time unless explicitly marked _vartime or comment to similar effect.
Field elements can be stack allocated.

Still todo

ECDSA verify tests
ECDH tests/bench
Exported/precomputed multiplication tables (for variable point mul + mul2)
Fallback support for non-hardwired curves

reneme · 2024-04-06T10:58:33Z

Awesome! I'll certainly have a look. Not before tomorrow, though.

randombit · 2024-04-06T11:08:31Z

No rush! This is certainly not going into 3.4 so there is plenty of time.

Most interesting thing about the sea of red in CI - GCC 11.4 on x86 fails with constexpr timeout, while same version of GCC on say Aarch64 or S390 accepts. So constexpr limits are architecture dependent [*] :( and thus intrinsically flaky since you can be near some limit without knowing it

[*] And also version dependent since GCC 13 on my machine is fine with the code without increasing the constexpr limit.

randombit · 2024-04-06T14:02:22Z

Clang seems to have a nasty bug where std::is_constant_evaluated returns false incorrectly in constexpr context, then it rejects the code because we used asm or volatile in constexpr context in the !std::is_constant_evaluated branch 😡

It works fine for me in Clang 17 on my machines so I'm assuming this was a Clang bug that was fixed subsequently. I'd vote for just bumping the Clang minimum version - that's an absolutely insane bug and nearly impossible to work around - but unfortunately the version in Android NDK also has the bug and we are stuck with that version at least until the next NDK release.

coveralls · 2024-04-06T14:15:06Z

coverage: 91.923% (-0.1%) from 92.024%
when pulling 43dfb1d on jack/pcurves
into adda2dd on master.

randombit · 2024-04-06T14:29:03Z

Downside of this approach is that (due to name mangling being somewhat horribly thought out for this case) object sizes are really big. Even with just 3 curves, on my machine the object file is one of the largest from the whole library, that will get a lot worse with 27. And compile times may prove untenable.

I'm sure 90% of this can be salvaged but I suspect in the end the string based instantiation won't work in practice 😭 which is a shame since it's quite pretty imo

randombit · 2024-04-06T14:32:39Z

Clang bug might be llvm/llvm-project#55638

randombit · 2024-04-06T14:36:27Z

This error

 call to consteval function 'Botan::p_minus<(unsigned char)'\x01', unsigned long, 4UL>' is not a constant expression

Looks like llvm/llvm-project#51182

randombit · 2024-04-06T15:27:56Z

I can repro the std::if_constant_evaluated problem using Clang 16 on my machine so it looks to be a bug fixed in Clang 17, though I don't see anything that looks relevant in the release notes.

reneme

Interesting! 😄

I merely skimmed through the changes and left a few suggestions and comments here and there. No thorough review whatsoever.

reneme · 2024-04-08T06:56:21Z

src/lib/pubkey/pcurves/pcurves_util.h

+*/
+
+#ifndef BOTAN_PCURVES_UTIL_H_
+#define BOTAN_PCURVES_UTIL_H_


General comment for this file (and perhaps other places):

I'd suggest to use std::span<const W, N> instead of std::array<>& for the parameters of those utilities. No need to restrict those functions to arrays, IMO. And a static-length span should provide the same guarantees and optimization opportunities as an array, no?

Main problem is for some technical reason I'm not bothering to learn the details of, C++ can't deduce the conversion from a statically sized array to a statically sized span. https://stackoverflow.com/questions/70983595

This can be worked around with some additional noise but the additional flexibility doesn't buy us anything in this context so I'd rather keep things as simple as possible.

src/lib/pubkey/pcurves/pcurves_util.h

src/lib/pubkey/pcurves/pcurves_impl.h

reneme · 2024-04-08T07:28:51Z

src/lib/pubkey/pcurves/pcurves_impl.h

+         auto v = bigint_monty_redc(m_val, Self::P, Self::P_dash);
+         std::reverse(v.begin(), v.end());
+         auto bytes = store_be(v);
+
+         if constexpr(Self::BYTES == Self::N * WordInfo<W>::bytes) {
+            return bytes;
+         } else {
+            // Remove leading zero bytes
+            const size_t extra = Self::N * WordInfo<W>::bytes - Self::BYTES;
+            std::array<uint8_t, Self::BYTES> out;
+            copy_mem(out.data(), &bytes[extra], Self::BYTES);
+            return out;
+         }


Perhaps we could get away without the if constexpr and the potential memory copy entirely?

Suggested change

auto v = bigint_monty_redc(m_val, Self::P, Self::P_dash);

std::reverse(v.begin(), v.end());

auto bytes = store_be(v);

if constexpr(Self::BYTES == Self::N * WordInfo<W>::bytes) {

return bytes;

} else {

// Remove leading zero bytes

const size_t extra = Self::N * WordInfo<W>::bytes - Self::BYTES;

std::array<uint8_t, Self::BYTES> out;

copy_mem(out.data(), &bytes[extra], Self::BYTES);

return out;

}

auto v = bigint_monty_redc(m_val, Self::P, Self::P_dash);

std::reverse(v.begin(), v.end());

std::array<uint8_t, Self::BYTES> out = {0};

static_assert(Self::N * WordInfo<W>::bytes >= Self::BYTES); // is this guranteed somehow anyway?

const size_t zero_padding_offset = Self::N * WordInfo<W>::bytes - Self::BYTES;

store_be(std::span{out}.template subspan<zero_padding_offset>(), v);

return out;

... this is untested. Just a sketch of the idea. store_be with a statically-sized out-param will static_assert that the byte buffer matches the input range exactly.

src/lib/pubkey/pcurves/pcurves_impl.h

reneme · 2024-04-08T07:43:03Z

src/lib/pubkey/pcurves/pcurves_impl.h

+      constexpr std::array<uint8_t, Self::BYTES> serialize() const {
+         std::array<uint8_t, Self::BYTES> r = {};
+         BufferStuffer pack(r);
+         pack.append(0x04);
+         pack.append(m_x.serialize());
+         pack.append(m_y.serialize());
+         return r;
+      }


It would be nice if we could use the concat helpers for this. Currently, that doesn't work, because concat requires the output type to have .insert().This could then also (statically) assert that the output array was filled entirely.

#3994 is work towards this. It makes concat() constexpr and also automagically inferring the output array size. So this should actually work as:

Suggested change

constexpr std::array<uint8_t, Self::BYTES> serialize() const {

std::array<uint8_t, Self::BYTES> r = {};

BufferStuffer pack(r);

pack.append(0x04);

pack.append(m_x.serialize());

pack.append(m_y.serialize());

return r;

}

constexpr std::array<uint8_t, Self::BYTES> serialize() const {

return concat(

store_le<uint8_t>(0x04), // store_le is just to make it look like a byte-array

m_x.serialize(),

m_y.serialize());

}

Something seems to be missing still

build/include/internal/botan/internal/loadstor.h: In instantiation of ‘constexpr auto Botan::store_le(ParamTs&& ...) [with ModifierT = unsigned char; ParamTs = {int}]’: build/include/internal/botan/internal/pcurves_impl.h:351:30: required from here build/include/internal/botan/internal/loadstor.h:699:67: error: no matching function for call to ‘store_any<Botan::detail::Endianness::Little, unsigned char>(int)’ 699 | return detail::store_any<detail::Endianness::Little, ModifierT>(std::forward<ParamTs>(params)...); | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

reneme · 2024-04-08T07:45:49Z

src/lib/pubkey/pcurves/pcurves_impl.h

+      void conditional_assign(bool cond, const Self& pt) {
+         m_x.conditional_assign(cond, pt.x());
+         m_y.conditional_assign(cond, pt.y());
+         m_z.conditional_assign(cond, pt.z());
+      }


I'm assuming this is needed for a constant-time implementation? So far, I saw it as an unspoken rule that all methods that are "supposed to be constant-time" are prefixed with ct_.... I found this a helpful convention, that might be applicable here as well.

In this code I'm going for the opposite convention namely that everything is constant time by default and anything that is intentionally variable time has a _vartime suffix.

(There are some current exceptions eg in the point multiplication that will be fixed before merging.)

randombit · 2024-04-08T15:43:09Z

GCC 11 miscompilation, excellent

This cuts the size of the object file by more than half, since we avoid embedding literal strings of the params into the final typenames.

Also some multiplication related improvements.

This drastically speeds up the projective->affine conversion since we can use a batch operation.

Improves point doubling performance by over 20% with Clang

randombit added this to the Botan 3.5.0 milestone Apr 6, 2024

randombit requested a review from reneme April 6, 2024 10:24

randombit force-pushed the jack/pcurves branch from e7b71fc to 2f7186e Compare April 6, 2024 13:47

randombit force-pushed the jack/pcurves branch from bcf81fe to 344cdd6 Compare April 6, 2024 14:43

randombit force-pushed the jack/pcurves branch 4 times, most recently from 3496464 to cd40575 Compare April 7, 2024 13:17

reneme reviewed Apr 8, 2024

View reviewed changes

randombit force-pushed the jack/pcurves branch 12 times, most recently from 2feb181 to 3bde391 Compare April 14, 2024 09:51

randombit added 29 commits May 5, 2024 16:21

Use a two step naming process for setting elliptic curve params

59daf66

This cuts the size of the object file by more than half, since we avoid embedding literal strings of the params into the final typenames.

Generalize integer representation, add P-521 reduction

8f87c5a

Take advantage of Montgomery friendly curves where p' is 1

91a16c3

Add a generic interface for scalars and points

b23aeb5

Also some multiplication related improvements.

In ECC mul speed test, always convert to affine

b71af1e

Implement scalar blinding and point rerandomization

058963f

Move the template/dynamic bridge to a header

c9b2ea9

More tests for pcurves

f1fbbad

Add more curves

e9ed0b8

Some arithmetic improvements

7ec1ea4

Use a fixed window for inversions, sqrt, etc

6b38f6a

Point multiplication improvements

0b878ce

Add a string to curve object helper

d04ac26

Add ECDSA signature test for pcurves

5520c7e

Avoid slow random scalars when field is not multiple of 8 bits

d3bab04

Add temporary ECDSA-pcurves speed test target

accaf3e

Use square when checking result of sqrt

e6257ce

Add a note regarding small scalar muls

5118c82

Add variable time mul2 (g*x+h*y) for pcurves

fb683a7

Avoid including infinity in the mul2 table

c7a04ae

This drastically speeds up the projective->affine conversion since we can use a batch operation.

Add speed test for pcurves ECDSA

3dd9013

Add support for precomputing tables for scalar multiplication

79ebff5

Remove redundant code in Montygomery reduction

b38dcf7

Tweak window sizes

b1e6b5a

Add mul2->x using table

bd00d52

Split pcurves to its own cmdlet in speed

0eab9b5

Special case doubling and mul by small integers

f6df10d

Improves point doubling performance by over 20% with Clang

Use the correct type

db991b0

Simplify initial loop of mul algorithms

43dfb1d

randombit force-pushed the jack/pcurves branch from cfe6c54 to 43dfb1d Compare May 5, 2024 20:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add library for compile time instantiation of elliptic curves #3979

Add library for compile time instantiation of elliptic curves #3979

randombit commented Apr 6, 2024 •

edited

reneme commented Apr 6, 2024

randombit commented Apr 6, 2024 •

edited

randombit commented Apr 6, 2024

coveralls commented Apr 6, 2024 •

edited

randombit commented Apr 6, 2024

randombit commented Apr 6, 2024

randombit commented Apr 6, 2024

randombit commented Apr 6, 2024

reneme left a comment

reneme Apr 8, 2024

randombit Apr 14, 2024

reneme Apr 8, 2024

reneme Apr 8, 2024 •

edited

reneme Apr 9, 2024

randombit Apr 14, 2024

reneme Apr 8, 2024

randombit Apr 8, 2024

randombit commented Apr 8, 2024

Add library for compile time instantiation of elliptic curves #3979

Are you sure you want to change the base?

Add library for compile time instantiation of elliptic curves #3979

Conversation

randombit commented Apr 6, 2024 • edited

reneme commented Apr 6, 2024

randombit commented Apr 6, 2024 • edited

randombit commented Apr 6, 2024

coveralls commented Apr 6, 2024 • edited

randombit commented Apr 6, 2024

randombit commented Apr 6, 2024

randombit commented Apr 6, 2024

randombit commented Apr 6, 2024

reneme left a comment

Choose a reason for hiding this comment

reneme Apr 8, 2024

Choose a reason for hiding this comment

randombit Apr 14, 2024

Choose a reason for hiding this comment

reneme Apr 8, 2024

Choose a reason for hiding this comment

reneme Apr 8, 2024 • edited

Choose a reason for hiding this comment

reneme Apr 9, 2024

Choose a reason for hiding this comment

randombit Apr 14, 2024

Choose a reason for hiding this comment

reneme Apr 8, 2024

Choose a reason for hiding this comment

randombit Apr 8, 2024

Choose a reason for hiding this comment

randombit commented Apr 8, 2024

randombit commented Apr 6, 2024 •

edited

randombit commented Apr 6, 2024 •

edited

coveralls commented Apr 6, 2024 •

edited

reneme Apr 8, 2024 •

edited