change semantics of `@memcpy` and `@memset` #15278

andrewrk · 2023-04-14T05:21:36Z

Now they use slices or array pointers with any element type instead of requiring byte pointers.

This is a breaking enhancement to the language.

std.mem.copy will be used for only possibly-overlapping slices, and thus renamed to copyForwards.
std.mem.set will be deleted in favor of the builtin.

Merge Checklist

Luukdegram · 2023-04-22T13:55:30Z

src/arch/wasm/CodeGen.zig

+    const value = try func.resolveInst(bin_op.rhs);
+    const len = switch (ptr_ty.ptrSize()) {
+        .Slice => try func.sliceLen(ptr),
+        .One => @as(WValue, .{ .imm64 = ptr_ty.childType().arrayLen() }),


All Wasm instructions are typed. Considering the length is of type usize, this should use imm32 on the wasm32 target, rather than imm64. (Same in airMemcpy).

Now they use slices or array pointers with any element type instead of requiring byte pointers. This is a breaking enhancement to the language. The safety check for overlapping pointers will be implemented in a future commit. closes #14040

* Sema: upgrade operands to array pointers if possible when emitting AIR. * Implement safety checks for length mismatch and aliasing. * AIR: make ptrtoint support slice operands. Implement in LLVM backend. * C backend: implement new `@memset` semantics. `@memcpy` is not done yet.

Also introduce memset_safe AIR tag and support it in C backend and LLVM backend.

Previously, this code casted the array pointer to u8 pointer, but I removed that in a different commit. This commit restores the cast, but instead of hard-coding u8, it uses the destination element pointer, since memset now supports arbitrary element types.

and avoid new language feature in std.ArrayList for now, until x86_64 self-hosted backend can implement it.

* make memset and memset_safe guarantee that if the length is comptime-known then it will be nonzero.

Needed due to the compiler depending on standard library APIs such as ArrayList that contain `@memset` and `@memcpy` calls in them. The number of parameters changed, so this is necessary for the compiler to build.

It does the same thing but has fewer bytes in the output.

store: The value to store may be undefined, in which case the destination memory region has undefined bytes after this instruction is evaluated. In such case ignoring this instruction is legal lowering. store_safe: Same as `store`, except if the value to store is undefined, the memory region should be filled with 0xaa bytes, and any other safety metadata such as Valgrind integrations should be notified of this memory region being undefined.

Previously it was not multiplying by the element ABI size. Now, it uses ptr_add instructions which do math based on the element type.

Pointer comparisons were triggering `-Wcompare-distinct-pointer-types` before this fix, which adds `(void*)` casts if the lhs type and rhs type do not match pointer sizeness.

This reverts commit 3c66850, reversing changes made to d0311e2.

Requires the minimum version to support ziglang/zig#15278

frmdstryr · 2023-05-10T22:40:34Z

Unfortunately, due to #13530 now any freestanding arm builds are completely unusable. Any undefined array causes a hard fault.

Release mode does not work either because compiler optimizes anything similar to a memset into the broken functions.

ziglang/zig#15278

* Add LIEF * Compile LIEF * Implement support for embedding files on macOS * proof of concept * Add zstd * Implement runtime support * Move some code around * Update .gitmodules * Upgrade zig ziglang/zig#15278 * leftover * leftover * delete dead code * Fix extname * Revert "Upgrade zig" This reverts commit dd968f3. * Revert "leftover" This reverts commit 7664de7. * Revert "leftover" This reverts commit 498005b. * various fixes * it works! * leftover * Make `zig build` a little faster * give up on code signing support * Support Linux & macOS * Finish removing LIEF * few more * Add zstd to list of deps * make it pretty --------- Co-authored-by: Jarred Sumner <709451+Jarred-Sumner@users.noreply.github.com>

andrewrk added enhancement Solving this issue will likely involve adding new logic or components to the codebase. breaking Implementing this issue could cause existing code to no longer compile or have different behavior. labels Apr 14, 2023

andrewrk force-pushed the memcpy-memset branch from 16b194d to 997476b Compare April 22, 2023 01:07

Luukdegram reviewed Apr 22, 2023

View reviewed changes

andrewrk force-pushed the memcpy-memset branch from ff9ed14 to de70914 Compare April 23, 2023 22:27

This was referenced Apr 23, 2023

llvm: store undef if safety is disabled #15003

Closed

remove memset calls when setting to undefined in unsafe modes #14104

Closed

andrewrk force-pushed the memcpy-memset branch 2 times, most recently from 92e8dd1 to c336d7f Compare April 25, 2023 04:41

andrewrk added 21 commits April 25, 2023 11:23

change semantics of @memcpy and @memset

a5c910a

Now they use slices or array pointers with any element type instead of requiring byte pointers. This is a breaking enhancement to the language. The safety check for overlapping pointers will be implemented in a future commit. closes #14040

C backend: implement new memcpy and inttoptr semantics

92186b8

wasm backend: implement new memcpy/memset and ptrtoint semantics

76e340c

x86 backend: implement new memcpy/memset semantics

25d1128

LLVM backend: support non-byte-sized memset

057c950

Also introduce memset_safe AIR tag and support it in C backend and LLVM backend.

Sema: implement comptime @memset

83a7303

Sema: implement comptime @memcpy

482a0f6

add behavior test for @memset on slices

7c56145

and avoid new language feature in std.ArrayList for now, until x86_64 self-hosted backend can implement it.

x86_64 backend: implement @memset for element ABI size > 1

881e931

* make memset and memset_safe guarantee that if the length is comptime-known then it will be nonzero.

x86_64 backend: support @memset with slices

58fabbc

langref: fix compile error

bd6f01c

update zig1.wasm

a8de15f

Needed due to the compiler depending on standard library APIs such as ArrayList that contain `@memset` and `@memcpy` calls in them. The number of parameters changed, so this is necessary for the compiler to build.

wasm backend: fix airMemset with slices

747f583

update test cases for new memcpy/memset semantics

1ba72bc

C backend: use ++ instead of += for airMemset

d604553

It does the same thing but has fewer bytes in the output.

Sema: fix memcpy alias safety incorrect math

792bbfa

Previously it was not multiplying by the element ABI size. Now, it uses ptr_add instructions which do math based on the element type.

zig fmt

401b7f6

C backend: fix lowering comparison when array ptr meets ptr

badad16

Pointer comparisons were triggering `-Wcompare-distinct-pointer-types` before this fix, which adds `(void*)` casts if the lhs type and rhs type do not match pointer sizeness.

andrewrk force-pushed the memcpy-memset branch from 1d7592c to badad16 Compare April 25, 2023 18:23

andrewrk merged commit 3c66850 into master Apr 26, 2023
10 checks passed

andrewrk deleted the memcpy-memset branch April 26, 2023 17:01

This was referenced Apr 26, 2023

implement @memcpy and @memset for tuples #15479

Open

actually use the new memory intrinsics #15481

Merged

truemedian added a commit to truemedian/zig that referenced this pull request Apr 27, 2023

Revert "Merge pull request ziglang#15278 from ziglang/memcpy-memset"

0f54035

This reverts commit 3c66850, reversing changes made to d0311e2.

truemedian mentioned this pull request Apr 27, 2023

Cannot build on zig master zigtools/zls#1158

Closed

sagehane added a commit to sagehane/zls that referenced this pull request Apr 29, 2023

build.zig: Update minimum Zig version

71a202f

Requires the minimum version to support ziglang/zig#15278

sagehane mentioned this pull request Apr 29, 2023

build.zig: Update minimum Zig version zigtools/zls#1159

Merged

binarycraft007 mentioned this pull request May 1, 2023

update to zig master: use @memset, @memcpy builtin ZigEmbeddedGroup/microzig#128

Merged

nektro mentioned this pull request May 2, 2023

update to zig master 0.11.0-dev.2861+3c66850e4 kivikakk/koino#46

Merged

paoda mentioned this pull request May 4, 2023

chore: update master version data zigtools/zls#1169

Merged

pkova mentioned this pull request May 7, 2023

tls: update finishRead2 for new @memcpy semantics #15612

Merged

DraagrenKirneh mentioned this pull request May 7, 2023

Zig build fails with deps URLs not ending in .tar.gz on GitHub in build.zig.zon #15419

Closed

Jarred-Sumner added a commit to oven-sh/bun that referenced this pull request May 13, 2023

Upgrade zig

dd968f3

ziglang/zig#15278

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

change semantics of `@memcpy` and `@memset` #15278

change semantics of `@memcpy` and `@memset` #15278

andrewrk commented Apr 14, 2023 •

edited

Luukdegram Apr 22, 2023

andrewrk Apr 23, 2023

frmdstryr commented May 10, 2023 •

edited

change semantics of @memcpy and @memset #15278

change semantics of @memcpy and @memset #15278

Conversation

andrewrk commented Apr 14, 2023 • edited

Merge Checklist

Luukdegram Apr 22, 2023

Choose a reason for hiding this comment

andrewrk Apr 23, 2023

Choose a reason for hiding this comment

frmdstryr commented May 10, 2023 • edited

change semantics of `@memcpy` and `@memset` #15278

change semantics of `@memcpy` and `@memset` #15278

andrewrk commented Apr 14, 2023 •

edited

frmdstryr commented May 10, 2023 •

edited