libc/machine/risc-v: Optimize string routines for misaligned pointers. - #19735
libc/machine/risc-v: Optimize string routines for misaligned pointers.#19735Fishwaldo wants to merge 2 commits into
Conversation
github-actions
Bot
added
Area: OS Components
|
| * Public Functions | ||
| ****************************************************************************/ | ||
|
|
||
| FAR char *strcpy(FAR char *dest, FAR const char *src) |
There was a problem hiding this comment.
The RISC-V machine directory optimized three functions and left the rest
to the generic C library. Two of the three were leaving most of their
speed on the table, and the functions that suffer most on this
architecture, the ones that take two pointers and have to cope with them
disagreeing about alignment, were not covered at all.
The two existing assembly routines learn what they most lacked. memcpy
becomes register-width aware, eight bytes a step on RV64 where it always
moved four, and gains a shifting path for the case it used to give up on:
when source and destination disagree about where a register boundary
falls, it now reads the two aligned words straddling each output word and
shifts them together, so no load and no store is ever misaligned and only
the head and tail go byte by byte. Its block loop is also reshaped, from
ten loads followed by ten stores into four groups of four loads and four
stores. That is the same instruction count in a different order, and it
is worth 31% on an aligned copy: ten loads to consecutive lines fill the
outstanding-miss capacity and the store burst that follows cannot overlap
the next iteration's loads.
strcmp stops treating every unaligned pointer as hopeless: two pointers
the same distance past a boundary are walked up to it bytewise and
compared a register at a time from there, which is the common case for
strings carved out of larger buffers. Only pointers that disagree about
the boundary keep the byte loop, because no single aligned load serves
both.
Six functions the directory did not cover are added as portable
word-at-a-time C, sharing one small header of the old tricks: memchr,
memcmp, memmove, strcpy, strncmp and strnlen.
Measured on 1.4 GHz silicon, twenty runs per configuration, MB/s with
95% confidence intervals. Three builds, named here by the option
that distinguishes them:
default neither option set, as most boards ship today
newlib CONFIG_LIBC_NEWLIB_OPTSPEED=y
riscv machine CONFIG_RISCV_STRING_FUNCTION=y, this work
default newlib riscv machine
memcpy aligned 412 +-1 4268 +-4 3981 +-4
memcpy mismatched 410 +-1 322 +-0 3073 +-3
memmove backward 439 +-0 439 +-0 3493 +-7
memcmp aligned 31 +-0 357 +-1 414 +-1
memcmp same offset 30 +-0 38 +-0 420 +-1
memchr 645 +-2 2489 +-30 2703 +-31
strncmp aligned 32 +-0 204 +-0 268 +-0
strncmp same offset 32 +-0 27 +-0 253 +-0
strcpy aligned 609 +-1 2110 +-14 1962 +-9
strcpy same offset 616 +-1 617 +-1 1813 +-8
The pattern is the one the architecture predicts. Where both pointers
are aligned newlib is already good, and beats this by 7% on memcpy and
strcpy. Where the two are merely consistent with each other, newlib
tests whether either pointer is aligned rather than whether the two
agree, and falls to a byte loop; these walk up to the boundary and
carry on a word at a time.
Correctness is not assumed: every function is exercised across source and
destination alignments zero through seven, twenty one sizes from zero up,
overlap in both directions for memmove, terminator placement and cap
interaction for the n-bounded functions, and guard bytes around every
destination. The same suite passes on RV32 under qemu, and it catches
deliberately injected corruption.
Assisted-by: Claude:claude-opus-5
Signed-off-by: Justin Hammond <justin@dynam.ac>
strlcpy earns its place by the numbers: forty six call sites in one kernel image, more than strcpy, and neither the generic library nor the arm64 directory offers an optimized one. The word loop keeps a byte in hand so the terminator always fits, and the length it must return regardless of truncation is finished by strlen. Measured over twenty runs at 1818 +-9 MB/s against the generic's 466 +-1, a byte pace. The two compare loops learn to carry one branch for several words: memcmp folds four words' differences together with XOR and OR before testing, strncmp two words' differences and terminators. Whatever stops the loop is then within a few words and the byte tail settles it. The honest measurement is that this helps less than it should, 343 to 420 MB/s for memcmp and 245 to 272 for strncmp, while the emitted loop is eight loads, four XORs, three ORs and a branch per thirty two bytes, and a single-stream word loop on this core runs at 3.3 GB/s. Something about two-stream reads here deserves a profile of its own; the loops are left unrolled because they are no worse anywhere and the shape is right once that is understood. Correctness for the new function: every source and destination alignment, lengths zero to twenty three, and every cap from zero to past the end. The return is always the source length, the result is terminated whenever the cap is nonzero, at most cap minus one bytes are copied, and a cap of zero writes nothing. Verified on RV64 silicon and RV32 under qemu alongside the whole existing suite. Assisted-by: Claude:claude-opus-5 Signed-off-by: Justin Hammond <justin@dynam.ac>
4a9f5c1 to
1fc8c83
Compare
|
Thanks - both comments land on the same question, and the answer to each is different.
I had missed the newlib implementations entirely, and went a long way down a rabbit hole on the assumption that only the default byte-at-a-time ones existed. Once I benchmarked against
#define UNALIGNED(x, y) \
(((long)(uintptr_t)(x) & (sizeof(long) - 1)) | ((long)(uintptr_t)(y) & (sizeof(long) - 1)))
if (!UNALIGNED(src0, dst0))Anything else copies the whole string a byte at a time. The RISC-V machine version instead asks whether the two pointers agree about where boundaries fall, walks up to the boundary bytewise, and takes words from there: if ((((uintptr_t)d ^ (uintptr_t)src) & (WORD_BYTES - 1)) == 0)For random pointers on RV64 the newlib condition holds about 1 in 64 times; this one about 1 in 8. And the common case in practice, strings carved out of the same larger buffer or a struct copied field by field, is exactly the one where both pointers share an offset but neither is aligned. Measured on 1.4 GHz RV64 silicon, twenty runs per configuration, MB/s with 95% confidence intervals. default is neither option set, newlib is With both pointers aligned, newlib is marginally ahead of the RISC-V version and I would not argue for replacing it on those numbers alone. With the pointers merely agreeing, newlib is at byte pace, 617 MB/s, and the RISC-V version is 1813. That second row is the entire justification. The same pattern decides the other retained routines:
Where newlib is genuinely competitive I have removed the RISC-V copy. Where it degrades to bytes on alignments that occur constantly, I have kept it. |
Fishwaldo
changed the title
|
@Fishwaldo but the c optimization routine need merge into bsd string implementation instead |
|
Hi @xiaoxiang781216 - I can shift the changes to the BSD version, but I You can benchmark with the test in Let me know if you still want me to shift the changes and there are others that can test on real hardware before we merge. (I did run some benchmarks on qemu, but had wide spread in results, so I don't have confidence in an emulated synthetic result!) |
we can verify your change on the different hardware.
yes, it is better to keep the general optimzaition out of the arch specific code.
do you switch qemu to icount mode? |
3 participants
Summary
libs/libc/machine/risc-voptimized three functions and left the rest to the C library. This improves two of those three and adds six more, chosen by measurement rather than by copying what other architectures happen to cover.The functions worth having here are the ones that take two pointers and must cope with them disagreeing about alignment. That is where the newlib implementations fall back to a byte loop, and where a RISC-V version can shift two aligned loads together instead.
Measured on 1.4 GHz RV64 silicon (ESWIN EIC7700X), twenty runs per configuration, MB/s with 95% confidence intervals. Three builds, named by the option that distinguishes them:
CONFIG_LIBC_NEWLIB_OPTSPEED=yCONFIG_RISCV_STRING_FUNCTION=y, this workRead the
same offsetandmismatchedrows againstnewlib. Those are pointer pairs that share an offset but are not word-aligned, strings carved out of a common buffer or structures copied field by field, and newlib is at byte pace on all of them. Onmemcpy mismatchedit is slower than the default byte loop, 322 against 410, because it pays for the alignment check and then falls back anyway.Where newlib is already competitive, this adds nothing and I have not touched those functions. An earlier revision of this PR also carried
strlen,strchr,strchrnulandstrrchr; measured against newlib they came out at 1.02x, 1.01x and 0.97x, so they are removed.The two existing assembly routines.
memcpybecomes register-width aware, eight bytes a step on RV64 where it always moved four, and gains a shifting path for the mismatched case it used to give up on. Its block loop is also reshaped from ten loads followed by ten stores into four groups of four loads and four stores, the same instruction count in a different order, worth 31% on an aligned copy, because ten loads to consecutive lines fill the outstanding-miss capacity and the store burst that follows cannot overlap the next iteration's loads.strcmpstops treating every unaligned pointer as hopeless.Six functions added as portable word-at-a-time C sharing one small header:
memchr,memcmp,memmove,strcpy,strncmp,strnlen. Plusstrlcpy, which has forty six call sites in one kernel image, more thanstrcpy, and which neither the C library nor the arm64 directory offers optimized: 1818 +-9 MB/s against 466 +-1.Impact
CONFIG_RISCV_STRING_FUNCTION=yonly, which isdefault n. Nothing changes for a build that does not set it.CONFIG_LIBC_NEWLIB_OPTSPEED; where both are set, the machine versions win for the functions covered here.Testing
Correctness for every function: every source and destination alignment, lengths zero through twenty three, and for the bounded functions every cap from zero to past the end. Verified on RV64 silicon and RV32 under qemu.
Performance as tabulated above, twenty runs per configuration; the harness verifies its own buffers and reports repetition counts.