Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*** buffer overflow detected *** with GCC 12 and -D_FORTIFY_SOURCE=3 #22801

Closed
marxin opened this issue Mar 21, 2022 · 22 comments · Fixed by #22917 or #25688
Closed

*** buffer overflow detected *** with GCC 12 and -D_FORTIFY_SOURCE=3 #22801

marxin opened this issue Mar 21, 2022 · 22 comments · Fixed by #22917 or #25688

Comments

@marxin
Copy link
Contributor

marxin commented Mar 21, 2022

Using the new -D_FORTIFY_SOURCE level, I see the following buffer overflow:

#0  __pthread_kill_implementation (threadid=<optimized out>, signo=signo@entry=6, no_tid=no_tid@entry=0) at pthread_kill.c:44
#1  0x00007ffff7aad1e3 in __pthread_kill_internal (signo=6, threadid=<optimized out>) at pthread_kill.c:78
#2  0x00007ffff7a5d306 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#3  0x00007ffff7a46813 in __GI_abort () at abort.c:79
#4  0x00007ffff7aa01b7 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff7be63cf "*** %s ***: terminated\n") at ../sysdeps/posix/libc_fatal.c:155
#5  0x00007ffff7b4530a in __GI___fortify_fail (msg=msg@entry=0x7ffff7be6375 "buffer overflow detected") at fortify_fail.c:26
#6  0x00007ffff7b438b6 in __GI___chk_fail () at chk_fail.c:28
#7  0x00007ffff7b43d3a in __read_chk (fd=<optimized out>, buf=<optimized out>, nbytes=<optimized out>, buflen=<optimized out>) at read_chk.c:30
#8  0x00007ffff78554a4 in read_virtual_file.constprop () at lib/libudev.so.1
#9  0x00007ffff784c9bf in device_read_uevent_file () at lib/libudev.so.1
#10 0x00007ffff784d026 in sd_device_get_devnum () at lib/libudev.so.1
#11 0x00007ffff785b7a8 in device_read_db_internal.constprop () at lib/libudev.so.1

It happens here:

systemd/src/basic/fileio.c

Lines 469 to 482 in e7949be

buf = malloc(size + 1);
if (!buf)
return -ENOMEM;
/* Use a bigger allocation if we got it anyway, but not more than the limit. */
size = MIN3(MALLOC_SIZEOF_SAFE(buf) - 1, max_size, READ_VIRTUAL_BYTES_MAX);
for (;;) {
ssize_t k;
/* Read one more byte so we can detect whether the content of the
* file has already changed or the guessed size for files from /proc
* wasn't large enough . */
k = read(fd, buf, size + 1);

where MALLOC_SIZEOF_SAFE is defined as:
/* This returns the number of usable bytes in a malloc()ed region as per malloc_usable_size(), in a way that
* is compatible with _FORTIFY_SOURCES. If _FORTIFY_SOURCES is used many memory operations will take the
* object size as returned by __builtin_object_size() into account. Hence, let's return the smaller size of
* malloc_usable_size() and __builtin_object_size() here, so that we definitely operate in safe territory by
* both the compiler's and libc's standards. Note that __builtin_object_size() evaluates to SIZE_MAX if the
* size cannot be determined, hence the MIN() expression should be safe with dynamically sized memory,
* too. Moreover, when NULL is passed malloc_usable_size() is documented to return zero, and
* __builtin_object_size() returns SIZE_MAX too, hence we also return a sensible value of 0 in this corner
* case. */
#define MALLOC_SIZEOF_SAFE(x) \
MIN(malloc_usable_size(x), __builtin_object_size(x, 0))

Even though, the comment counts with -D_FORTIFY_SOURCE it behaves wrongly as SIZE_MAX is returned for an unknown object. Thus we end up with size == malloc_usable_size and the assert happens.

There's a reduced test-case:

#include <malloc.h>
#include <inttypes.h>
#include <fcntl.h>
#include <unistd.h>

#define MIN(a, b) (a < b ? a : b)

size_t size = 4096;

int main(int argc, char **argv)
{
  void *buf = malloc (size);
  if (buf == 0)
    return 1;

  size_t usable = MIN(malloc_usable_size(buf), __builtin_object_size(buf, 0));
  int fd = open("a.out", 0);
  read(fd, buf, usable);
}
gcc m.c -D_FORTIFY_SOURCE=3 -O2 && ./a.out 
m.c: In function ‘main’:
m.c:18:3: warning: ignoring return value of ‘read’ declared with attribute ‘warn_unused_result’ [-Wunused-result]
   18 |   read(fd, buf, usable);
      |   ^~~~~~~~~~~~~~~~~~~~~
*** buffer overflow detected ***: terminated
Aborted (core dumped)

I think one possible fix would be usage of __builtin_dynamic_object_size, or not
using malloc_usable_size at all.

CCing GCC developer of the new fortification level @siddhesh.

@poettering
Copy link
Member

I am not sure I follow, but aren't you just saying that _FORTIFY_SOURCE=3 is just broken, and disagrees with malloc_usable_size(), i.e. as opposed to the function's name assumes the allocated space is not actually usable to the end, if larger than what was originally allocated?

If so, why is this a systemd bug and not a gcc bug?

@marxin
Copy link
Contributor Author

marxin commented Mar 21, 2022

I am not sure I follow, but aren't you just saying that _FORTIFY_SOURCE=3 is just broken, and disagrees with malloc_usable_size(), i.e. as opposed to the function's name assumes the allocated space is not actually usable to the end, if larger than what was originally allocated?

Yeah, it's a grey area and I bet Siddhesh can take a look at it.

Just reading the malloc_usable_size documentation:

...
Athough the excess bytes can be overwritten by the application without ill effects, this is not good pro-
gramming practice: the number of excess bytes in an allocation depends on the underlying implementation.

The main use of this function is for debugging and introspection.

So the question is if you really need the functionality as it's intended for debugging or introspection purposes?

@poettering
Copy link
Member

poettering commented Mar 21, 2022

So the question is if you really need the functionality as it's intended for debugging or introspection purposes?

it makes life a lot more pleasant to use this. And we use it all over the place, for example for things like strextend() and stuff.

There were some other checkers like this in the past that had similar issues like this. But we got reassurances from libc people that malloc_usable_size() means what it says, and our usage is safe:

#19203 (comment)

@siddhesh
Copy link
Contributor

siddhesh commented Mar 22, 2022

Not all malloc implementations implement malloc_usable_size since it's not a required interface, so emitting it in place of __builtin_dynamic_object_size by default would either be flaky or expensive.

However, given that _FORTIFY_SOURCE results can now be expressions and not just constants, there's now scope in future for a -fuse-malloc-usable-size that emits a malloc_usable_size call but that needs upstream gcc buy-in. Until then, it seems like systemd usage of residual size using malloc_usable_size is incompatible with _FORTIFY_SOURCE, not just level 3.

@marxin
Copy link
Contributor Author

marxin commented Mar 24, 2022

That said, will you accept @poettering a pull request where we'll use __builtin_dynamic_object_size (instead of __builtin_object_size) if:
#if _FORTIFY_SOURCE >= 3?

@poettering
Copy link
Member

why would we ever want to use __builtin_object_size() if __builtin_dynamic_object_size() is available here? i.e. shouldn't we just switch unconditionally if the new builtin exists? why conditoinalize on _FORTIFY_SOURCE at all regarding its use?

@marxin
Copy link
Contributor Author

marxin commented Mar 24, 2022

Yes, I would do that. Can you please make such a change?

marxin added a commit to marxin/systemd that referenced this issue Mar 31, 2022
As explained in the issue, -D_FORTIFY_SOURCE=3 requires usage
of __builtin_dynamic_object_size in MALLOC_SIZEOF_SAFE macro.

Fixes: systemd#22801
marxin added a commit to marxin/systemd that referenced this issue Mar 31, 2022
As explained in the issue, -D_FORTIFY_SOURCE=3 requires usage
of __builtin_dynamic_object_size in MALLOC_SIZEOF_SAFE macro.

Fixes: systemd#22801
yuwata pushed a commit that referenced this issue Mar 31, 2022
As explained in the issue, -D_FORTIFY_SOURCE=3 requires usage
of __builtin_dynamic_object_size in MALLOC_SIZEOF_SAFE macro.

Fixes: #22801
gentoo-bot pushed a commit to gentoo/gentoo that referenced this issue Apr 28, 2022
Notably not bothering to revbump for now because this manifests
during self-execution during build and FORTIFY_SOURCE=3 is only
available in GCC 12 which isn't even released yet, let alone
exposed or enabled by default in Gentoo.

It's far more likely that systemd 251 will be released (or
at least another RC for it) before we're even close to unleashing
FORTIFY_SOURCE=3 on Gentoo Hardened users by default.

Bug: systemd/systemd#22801
Signed-off-by: Sam James <sam@gentoo.org>
@evverx
Copy link
Member

evverx commented Jun 3, 2022

I've just opened #23621 where I reverted this patch. I'm not sure how to make systemd work with gcc and _FORTIFY_SOURCE=3 but calling __builtin_dynamic_object_size seems too risky to me.

evverx added a commit to evverx/systemd that referenced this issue Jun 4, 2022
…_size."

This reverts commit 0bd2925.

Unlike __builtin_object_size, __builtin_dynamic_object_size is called at
runtime and it isn't guaranteed anywhere that it always works
with every pointer passed to it. It currently works with gcc because it
returns -1 most of the time (which means that malloc_usable_size
is used more often than not) but with clang (and probably gcc in the
foreseeable future) it's just not safe to assume that all pointers
can be handled at runtime.

Closes systemd#23619 and
systemd#23150.

Reopens systemd#22801
evverx added a commit to evverx/systemd that referenced this issue Jun 4, 2022
…_size."

This reverts commit 0bd2925.

Unlike __builtin_object_size, __builtin_dynamic_object_size is called at
runtime and it isn't guaranteed anywhere that it always works
with every pointer passed to it. It currently works with gcc because it
returns -1 most of the time (which means that malloc_usable_size
is used more often than not) but with clang (and probably gcc in the
foreseeable future) it's just not safe to assume that all pointers
can be handled at runtime.

Closes systemd#23619 and
systemd#23150.

Reopens systemd#22801
evverx added a commit to evverx/systemd that referenced this issue Jun 5, 2022
…_size."

This reverts commit 0bd2925.

Unlike __builtin_object_size, __builtin_dynamic_object_size is called at
runtime and it isn't guaranteed anywhere that it always works
with every pointer passed to it. It currently works with gcc because it
returns -1 most of the time (which means that malloc_usable_size
is used more often than not) but with clang (and probably gcc in the
foreseeable future) it's just not safe to assume that all pointers
can be handled at runtime.

Closes systemd#23619 and
systemd#23150.

Reopens systemd#22801
evverx added a commit to evverx/systemd that referenced this issue Jun 5, 2022
…_size."

This reverts commit 0bd2925.

It isn't guaranteed anywhere that __builtin_dynamic_object_size can
always deduce the size of every object passed to it so systemd
can end up using either malloc_usable_size or
__builtin_dynamic_object_size when pointers are passed around,
which in turn can lead to actual segfaults like the one mentioned in
systemd#23619.

Apparently __builtin_object_size can return different results for
pointers referring to the same memory as well but somehow it hasn't
caused any issues yet. Looks like this whole
malloc_usable_size/FORTIFY_SOURCE stuff should be revisited.

Closes systemd#23619 and
systemd#23150.

Reopens systemd#22801
@keszybz keszybz reopened this Jun 6, 2022
@yuwata yuwata added this to the v252 milestone Jun 9, 2022
@poettering poettering modified the milestones: v252, v253 Jul 26, 2022
bluca pushed a commit to bluca/systemd-stable that referenced this issue Nov 7, 2022
…_size."

This reverts commit 0bd2925.

It isn't guaranteed anywhere that __builtin_dynamic_object_size can
always deduce the size of every object passed to it so systemd
can end up using either malloc_usable_size or
__builtin_dynamic_object_size when pointers are passed around,
which in turn can lead to actual segfaults like the one mentioned in
systemd/systemd#23619.

Apparently __builtin_object_size can return different results for
pointers referring to the same memory as well but somehow it hasn't
caused any issues yet. Looks like this whole
malloc_usable_size/FORTIFY_SOURCE stuff should be revisited.

Closes systemd/systemd#23619 and
systemd/systemd#23150.

Reopens systemd/systemd#22801

(cherry picked from commit 2cfb790)
bluca pushed a commit to systemd/systemd-stable that referenced this issue Nov 7, 2022
…_size."

This reverts commit 0bd2925.

It isn't guaranteed anywhere that __builtin_dynamic_object_size can
always deduce the size of every object passed to it so systemd
can end up using either malloc_usable_size or
__builtin_dynamic_object_size when pointers are passed around,
which in turn can lead to actual segfaults like the one mentioned in
systemd/systemd#23619.

Apparently __builtin_object_size can return different results for
pointers referring to the same memory as well but somehow it hasn't
caused any issues yet. Looks like this whole
malloc_usable_size/FORTIFY_SOURCE stuff should be revisited.

Closes systemd/systemd#23619 and
systemd/systemd#23150.

Reopens systemd/systemd#22801

(cherry picked from commit 2cfb790)
@siddhesh
Copy link
Contributor

siddhesh commented Dec 8, 2022

FYI, I started a conversation on malloc_usable_size here:

https://sourceware.org/pipermail/libc-alpha/2022-November/143599.html

which continued here:

https://sourceware.org/pipermail/libc-alpha/2022-December/143667.html

where we discuss systemd's use of malloc_usable_size and if we can support it in an allocator. In summary, here's the current consensus:

  • Using extra space got from malloc_usable_size is unsafe because while the interface returns the currently usable size for an object, there’s no guarantee that the size will be constant for the lifetime of the object because the allocator is free to, for example, change the usable size to optimize workloads.
  • The man page currently states that using the extra space reported by malloc_usable_size is bad programming practice and should be avoided. There’s consensus in the glibc community to make that warning stronger, making it clear that actually using any size reported by malloc_usable_size is unsafe. The "without ill effects" phrasing there is not really true in the general case, it's only true for the current state of glibc malloc and it's not a guarantee that can be reasonably maintained.
  • There is also a tentative consensus to deprecate malloc_usable_size since its utility is limited and since it only promotes unsafe programming practices. I'm going to propose adding a link time warning in glibc 2.37 and eventually disallow linking in a future release.

I think the safest way forward here for systemd is to wean itself away from malloc_usable_size and explicitly pass object sizes from the point that they’re allocated. This is not just to get it working with _FORTIFY_SOURCE, but also to avoid unsafe behaviour in general because systemd is currently too optimistic about how malloc_usable_size could behave.

@poettering
Copy link
Member

When we asked the last time glibc people said everything is fine and we can rely on it. Really sucks if you then suddenly deprecate it without replacement. Why not just make it return the actually safe size? Then everything would just work. We are not using this just as an optimization, we are using it because it makes many things so kuch easier (for example a free() wrapper call that erases the memory). For that it would be entirely fine to just make the call return the exact size originally passed. But just breaking things without replacement after we explicitly asked about it earlier and where our use was okeyed is just mean.

@poettering
Copy link
Member

To say this explicitly: we really dont care about using the extra space malloc_usable_size() returns. All we care about is knowing how much at least to erase. Give us that and we are happy.

@siddhesh
Copy link
Contributor

siddhesh commented Dec 9, 2022

When we asked the last time glibc people said everything is fine and we can rely on it.

There was a misunderstanding about how systemd was using malloc_usable_size (I realized late that it's not the extra size you're after) and on fixing that misunderstanding, we stumbled upon the revelation that malloc_usable_size return values could vary over the lifetime of an object and systemd's usage could stumble over that.

Really sucks if you then suddenly deprecate it without replacement. Why not just make it return the actually safe size? Then everything would just work. We are not using this just as an optimization, we are using it because it makes many things so kuch easier (for example a free() wrapper call that erases the memory). For that it would be entirely fine to just make the call return the exact size originally passed.

Unfortunately, the size is currently not stored anywhere and doing so would result in an allocation overhead for all applications. Also, while this may fix it for glibc, the key problem is that the interface does not guarantee that the return value from malloc_usable_size will be constant over the lifetime of an object, so the current usage is not portable in that context.

But just breaking things without replacement after we explicitly asked about it earlier and where our use was okeyed is just mean.

I can assure you that's not the spirit with which we're doing this. I can't speak for previous endorsements because I didn't make them, but AFAICT, they don't seem to speak for the design pattern at all, they talk more about the allocator interface contract re. malloc_usable_size. AFAICT, the systemd usage of malloc_usable_size was properly understood only recently within the glibc community.

I've been trying to think of alternatives since this bug was filed (see siddhesh@e6760d4 or the patch I posted in my email to libc-alpha for example, which could help fix the undefined behaviour issue but not the issue of arbitrarily changing chunk sizes) and really the cleanest and most portable way I can think of is for systemd to start passing object sizes around instead of relying on malloc_usable_size to get it. You'll probably see a minor performance improvement too due to the elimination of all those PLT jumps.

@poettering
Copy link
Member

I mean, come on, maintaining the size in systemd means basically putting a second layer of memory management on top of glibcs. If that's what we have to do, I think I am more tempted to simply not use glibc's allocator at all anymore. I mean, maintaining our own metadata for every allocation is just nuts.

I mean, it appears to me that you have your FORTIFY_SOURCE=3 thing as a goal and that's the only thing that matters, and you really don#t care about anyone actually using this the stuff, i.e. our usecase is just not interesting to you.

I mean, please understand: we don't really care about the size per se. What we care about most importantly is memory allocations which get automatically erase on free(). We can certainly give up on that idea, on the altar of FORTIFY_SOURCE=3, but it should be very clear that this is a serious degradation of security, and in theory on the altar of FORTIFY_SOURCE=3.

Anyway, what about putting usability in the foreground, and all security instead of throwing it all out of the window for a much less interesting concept of security, i.e. FORTIFY_SOURCE=3?

@poettering
Copy link
Member

Also, can you explain to me, why it's such a terrible idea to just make malloc_usable_size() work as one would expect? I mean, apparently FORITFY_SOURCE=3 has a supposedly good idea of the expected size of memory allocations. So why in heaven can't malloc_usable_size() return a size that takes that information into account?

Why do you think it's a good idea to know the right size, but under no circumstances allowing code to actually make use of that information safely? I mean, what's driving this?

@siddhesh
Copy link
Contributor

siddhesh commented Dec 9, 2022

I mean, come on, maintaining the size in systemd means basically putting a second layer of memory management on top of glibcs. If that's what we have to do, I think I am more tempted to simply not use glibc's allocator at all anymore. I mean, maintaining our own metadata for every allocation is just nuts.

I doubt that, systemd seems to be the only application in all of the distribution (that I tested and evidently, Suse too) that seems to trip on this and surely everyone doesn't have their own allocator. Passing an array along with its size doesn't mean managing ones own allocations.

I mean, it appears to me that you have your FORTIFY_SOURCE=3 thing as a goal and that's the only thing that matters, and you really don#t care about anyone actually using this the stuff, i.e. our usecase is just not interesting to you.

The systemd use case is that it is risky for reasons that I already explained a couple of times above. Whether it is interesting to me or not is not really the point because I'm not personally invested in seeing malloc_usable_size go away. I see deprecation as the best option because in the larger ecosystem I don't see any way to implement it safely enough within the current API contracts for a general purpose allocator.

I mean, please understand: we don't really care about the size per se. What we care about most importantly is memory allocations which get automatically erase on free(). We can certainly give up on that idea, on the altar of FORTIFY_SOURCE=3, but it should be very clear that this is a serious degradation of security, and in theory on the altar of FORTIFY_SOURCE=3.

Let me look at that more closely and see if there's anything I can help with.

Anyway, what about putting usability in the foreground, and all security instead of throwing it all out of the window for a much less interesting concept of security, i.e. FORTIFY_SOURCE=3?

I'm not sure how keeping malloc_usable_size in would improve 'all' security. If you're talking about clearing memory on free, again, it's not a use case unique to systemd and I'm sure there are ways to solve it that don't require using malloc_usable_size.

Also, can you explain to me, why it's such a terrible idea to just make malloc_usable_size() work as one would expect? I mean, apparently FORITFY_SOURCE=3 has a supposedly good idea of the expected size of memory allocations. So why in heaven can't malloc_usable_size() return a size that takes that information into account?

It's because they're two different sources of truth, one from allocator internals and another from whatever the compiler can glean from the code it has visibility to. Also, making malloc_usable_size return the precise size would mean overhead for all applications and something I'd have to work on proving as being acceptable across the ecosystem. All this for one user of the allocator, i.e. systemd.

Why do you think it's a good idea to know the right size, but under no circumstances allowing code to actually make use of that information safely? I mean, what's driving this?

It's not a good idea, that's why it's being deprecated. I don't have the context on when the API was added, but given the programming practice warning, it never seems to have been a desired intent.

@keszybz
Copy link
Member

keszybz commented Dec 9, 2022

  • There is also a tentative consensus to deprecate malloc_usable_size since its utility is limited and since it only promotes unsafe programming practices.

Apart from erasing before free, we also use malloc_usable_size() when growing buffers (see our GREEDY_REALLOC helper). Before, we explicitly maintained a separate size_t n_allocated only for the purposes of the occasional call to realloc(). By using malloc_usable_size() we avoid that and can pass around a single pointer instead of a pointer and a size, and things are generally nicer. Size bookkeeping was also a source of mistakes, e.g. when people confused size in bytes with the number of objects. So I think it's fair that "promotes unsafe programming practices" is not true. If we have a working malloc_usable_size() or something equivalent, we make memory management more automatic, and mistakes in memory management are a well-known source of significant bugs.

@siddhesh
Copy link
Contributor

siddhesh commented Dec 9, 2022

  • There is also a tentative consensus to deprecate malloc_usable_size since its utility is limited and since it only promotes unsafe programming practices.

Apart from erasing before free, we also use malloc_usable_size() when growing buffers (see our GREEDY_REALLOC helper).

Is that for performance? I've added a lockless short-circuit to make that path faster in glibc malloc, so that shouldn't be necessary anymore.

Before, we explicitly maintained a separate size_t n_allocated only for the purposes of the occasional call to realloc(). By using malloc_usable_size() we avoid that and can pass around a single pointer instead of a pointer and a size, and things are generally nicer. Size bookkeeping was also a source of mistakes, e.g. when people confused size in bytes with the number of objects.

That's a fair point.

So I think it's fair that "promotes unsafe programming practices" is not true. If we have a working malloc_usable_size() or something equivalent, we make memory management more automatic, and mistakes in memory management are a well-known source of significant bugs.

We've talked about a new interface that returns the actually allocated size, there are standards proposals on those lines for C++ too. I suppose to support something like that we'll need to keep a consistent internal record of size somewhere anyway, that we can then pass out to callers. Let me think about this.

@poettering
Copy link
Member

I mean, come on, maintaining the size in systemd means basically putting a second layer of memory management on top of glibcs. If that's what we have to do, I think I am more tempted to simply not use glibc's allocator at all anymore. I mean, maintaining our own metadata for every allocation is just nuts.

I doubt that, systemd seems to be the only application in all of the distribution (that I tested and evidently, Suse too) that seems to trip on this and surely everyone doesn't have their own allocator. Passing an array along with its size doesn't mean managing ones own allocations.

So you are concerned about two sources of truth, but here you want to push everyone using your APIs to do this (i.e. malloc's own, and the user's assumption of the size), and that's risky as it could be. That's bad security engineering: making things complicated for users to use correctly, and easy to fuck up.

Seriously, this just creates the desire to give up on this mess and start already with switching finally over to Rust.

@poettering
Copy link
Member

The systemd use case is that it is risky for reasons that I already explained a couple of times above. Whether it is interesting to me or not is not really the point because I'm not personally invested in seeing malloc_usable_size go away. I see deprecation as the best option because in the larger ecosystem I don't see any way to implement it safely enough within the current API contracts for a general purpose allocator.

I don't follow. Why is this so problematic? Why can't make gcc make its knowledge available so that glibc can incorporate it into what it reports in malloc_usable_size()?

I'm not sure how keeping malloc_usable_size in would improve 'all' security. If you're talking about clearing memory on free, again, it's not a use case unique to systemd and I'm sure there are ways to solve it that don't require using malloc_usable_size.

btw, what we actually want is some flag we can set on allocations that says "this is for sensitive" data, and that puts the allocation in locked memory and erase it on free automatically. I suggested this to glibc people somewhere (on some bugzilla, don't remember), but this was shot down, telling me to use openssl's secmem stuff instead, but dealing with two allocators in the same program is just a mess, and would probably result in security relevant bugs all the time. it's so weird that there's zero interest from glibc people to solve these — very important as I believe – things, but a lot of interest in breaking the approaches people find to work around them.

It's because they're two different sources of truth, one from allocator internals and another from whatever the compiler can glean from the code it has visibility to.

Why would two sources of truth actually have to be an issue? First of all, we already take the minimum of malloc_usable_size() and __builtin_object_size() for all our uses into account. We'd also use the dynamic version of the latter if it would actually work. Alas it appears to be fucked up, see that other PR.

But more importantly: why doesn't glibc do this on its own? malloc_usable_size() could take this stuff into account too, if it wanted.

BTW, I looked into replacing our use of malloc_usable_size() with a combination of malloc_usable_size() + realloc() so that we officialy tell glibc that we actually might write to the rest of the buffer. But when I toyed around it I realized glibc was lying and ended up copying the memory in this case sometimes.

But if you'd fix that in glibc, i.e. that we get the guarantee that realloc(p, malloc_usable_size(p)) == p always holds, then we could do this, and gcc could grok this too.

Also, making malloc_usable_size return the precise size would mean overhead for all applications and something I'd have to work on proving as being acceptable across the ecosystem. All this for one user of the allocator, i.e. systemd.__builtin_object_size

Maybe try to turn this around: would it be good if more people would use malloc_usable_size() properly like we do? I think absolutely, yes, it would be. It improves security, and robustness, it removes secondary sources of truth, secondary memory management bookkeeping. So instead of just making clear to us that systemd is not reason enough to care to improve the situation, how about thinking what can you do to make programming with C safer and easier in general? having malloc_usable_size() that works would certainly achieve that.

@siddhesh
Copy link
Contributor

siddhesh commented Dec 9, 2022

The systemd use case is that it is risky for reasons that I already explained a couple of times above. Whether it is interesting to me or not is not really the point because I'm not personally invested in seeing malloc_usable_size go away. I see deprecation as the best option because in the larger ecosystem I don't see any way to implement it safely enough within the current API contracts for a general purpose allocator.

I don't follow. Why is this so problematic? Why can't make gcc make its knowledge available so that glibc can incorporate it into what it reports in malloc_usable_size()?

gcc gleans object sizes from the allocation points that it is able to see, i.e. the allocation has to be within the same translation unit and somehow associated with the pointer in the function that the compiler is currently building.

For malloc_usable_size to be able to see the object size, it will need to have context of the caller so that it can see the same object size as the compiler would in the caller. That is currently not possible and I don't think there's a way to actually achieve that.

I'm not sure how keeping malloc_usable_size in would improve 'all' security. If you're talking about clearing memory on free, again, it's not a use case unique to systemd and I'm sure there are ways to solve it that don't require using malloc_usable_size.

btw, what we actually want is some flag we can set on allocations that says "this is for sensitive" data, and that puts the allocation in locked memory and erase it on free automatically. I suggested this to glibc people somewhere (on some bugzilla, don't remember), but this was shot down, telling me to use openssl's secmem stuff instead, but dealing with two allocators in the same program is just a mess, and would probably result in security relevant bugs all the time. it's so weird that there's zero interest from glibc people to solve these — very important as I believe – things, but a lot of interest in breaking the approaches people find to work around them.

I suppose something like MALLOC_PERTURB_ could be used to erase freed objects, but if you need it for specific blocks, it'll have to be an extension. If you can find the bug, I'm interested in knowing the rationale for the rejection.

Contexts for requests change over years and maybe what was deemed infeasible then could be worth revisiting today.

It's because they're two different sources of truth, one from allocator internals and another from whatever the compiler can glean from the code it has visibility to.

Why would two sources of truth actually have to be an issue? First of all, we already take the minimum of malloc_usable_size() and __builtin_object_size() for all our uses into account. We'd also use the dynamic version of the latter if it would actually work. Alas it appears to be fucked up, see that other PR.

So the __builtin_object_size hack works because that builtin is only able to see static allocations within a function (including inlines) and it becomes a single static value, in most cases, just bails out with a -1. __builtin_dynamic_object_size on the other hand comes up with a precise expression and not only that, it succeeds far more than __builtin_object_size. More precisely, where __builtin_object_size succeeds ~9% of the time in systemd, it succeeds over 30% of the time with __builtin_dynamic_object_size.

In the systemd context, it is unlikely for one function to see an object size as the result of __builtin_object_size and another to see it as malloc_usable_size, whereas it's more likely for that discrepancy to crop up with __builtin_dynamic_object_size. This is why the replacement with __builtin_dynamic_object_size broke the hack.

BTW, I looked into replacing our use of malloc_usable_size() with a combination of malloc_usable_size() + realloc() so that we officialy tell glibc that we actually might write to the rest of the buffer. But when I toyed around it I realized glibc was lying and ended up copying the memory in this case sometimes.

You can do that without calling realloc, see my PR #25688

But if you'd fix that in glibc, i.e. that we get the guarantee that realloc(p, malloc_usable_size(p)) == p always holds, then we could do this, and gcc could grok this too.

I fixed that yesterday but it's not really necessary for systemd: https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=f4f2ca1509288f6f780af50659693a89949e7e46

@keszybz
Copy link
Member

keszybz commented Dec 9, 2022

  • Using extra space got from malloc_usable_size is unsafe because while the interface returns the currently usable size for an object, there’s no guarantee that the size will be constant for the lifetime of the object because the allocator is free to, for example, change the usable size to optimize workloads.

So… why not just cut this short and not allow the allocator to ever change the usable size down? If "ever" is not possible, maybe make malloc_usable_size set a flag on the allocation when it's called, and not allow the allocator to shrink those allocations? Most applications don't call malloc_usable_size, to there'd be no difference for them, and the ones that do could use the returned value safely.

@siddhesh
Copy link
Contributor

siddhesh commented Dec 9, 2022

  • Using extra space got from malloc_usable_size is unsafe because while the interface returns the currently usable size for an object, there’s no guarantee that the size will be constant for the lifetime of the object because the allocator is free to, for example, change the usable size to optimize workloads.

So… why not just cut this short and not allow the allocator to ever change the usable size down? If "ever" is not possible, maybe make malloc_usable_size set a flag on the allocation when it's called, and not allow the allocator to shrink those allocations? Most applications don't call malloc_usable_size, to there'd be no difference for them, and the ones that do could use the returned value safely.

Shrinking is not so much a problem, given that the malloc_usable_size man page says the memory can be accessed without ill effects (as far as the allocator is concerned), the size will remain at least the value returned by malloc_usable_size. It could however grow and that would again result in different parts of the program seeing different sizes at different points in time.

However, that's not the current behaviour; right now glibc malloc right now does not touch allocated blocks, so the usable size will remain the same until someone finds some performance opportunity there in future and then it all breaks again. After mulling a bit, I finally posted PR #25688 (looks like it has failed CI, I need to look at fixing that) because it fixes at least one problem, i.e. malloc_usable_size use with the current implementation of glibc malloc_usable_size while we continue discussing its fate in glibc. FWIW, it won't be an immediate deprecation, we tend to take years between declaring something as deprecated and actually removing the default symbol.

poettering pushed a commit that referenced this issue Dec 14, 2022
systemd uses malloc_usable_size() everywhere to use memory blocks
obtained through malloc, but that is abuse since the
malloc_usable_size() interface isn't meant for this kind of use, it is
for diagnostics only.  This is also why systemd behaviour is flaky when
built with _FORTIFY_SOURCE.

One way to make this more standard (and hence safer) is to, at every
malloc_usable_size() call, also 'reallocate' the block so that the
compiler can see the larger size.  This is done through a dummy
reallocator whose only purpose is to tell the compiler about the larger
usable size, it doesn't do any actual reallocation.

Florian Weimer pointed out that this doesn't solve the problem of an
allocator potentially growing usable size at will, which will break the
implicit assumption in systemd use that the value returned remains
constant as long as the object is valid.  The safest way to fix that is
for systemd to step away from using malloc_usable_size() like this.

Resolves #22801.
Yamakuzure pushed a commit to elogind/elogind that referenced this issue Dec 23, 2022
…_size."

This reverts commit 0bd292567a543d124cd303f7dd61169a209cae64.

It isn't guaranteed anywhere that __builtin_dynamic_object_size can
always deduce the size of every object passed to it so systemd
can end up using either malloc_usable_size or
__builtin_dynamic_object_size when pointers are passed around,
which in turn can lead to actual segfaults like the one mentioned in
systemd/systemd#23619.

Apparently __builtin_object_size can return different results for
pointers referring to the same memory as well but somehow it hasn't
caused any issues yet. Looks like this whole
malloc_usable_size/FORTIFY_SOURCE stuff should be revisited.

Closes systemd/systemd#23619 and
systemd/systemd#23150.

Reopens systemd/systemd#22801
bluca pushed a commit to bluca/systemd that referenced this issue Jan 27, 2023
systemd uses malloc_usable_size() everywhere to use memory blocks
obtained through malloc, but that is abuse since the
malloc_usable_size() interface isn't meant for this kind of use, it is
for diagnostics only.  This is also why systemd behaviour is flaky when
built with _FORTIFY_SOURCE.

One way to make this more standard (and hence safer) is to, at every
malloc_usable_size() call, also 'reallocate' the block so that the
compiler can see the larger size.  This is done through a dummy
reallocator whose only purpose is to tell the compiler about the larger
usable size, it doesn't do any actual reallocation.

Florian Weimer pointed out that this doesn't solve the problem of an
allocator potentially growing usable size at will, which will break the
implicit assumption in systemd use that the value returned remains
constant as long as the object is valid.  The safest way to fix that is
for systemd to step away from using malloc_usable_size() like this.

Resolves systemd#22801.

(cherry picked from commit 7929e18)
(cherry picked from commit 34b9edd)
(cherry picked from commit 70653eb)
eworm-de pushed a commit to eworm-de/systemd that referenced this issue Feb 4, 2023
systemd uses malloc_usable_size() everywhere to use memory blocks
obtained through malloc, but that is abuse since the
malloc_usable_size() interface isn't meant for this kind of use, it is
for diagnostics only.  This is also why systemd behaviour is flaky when
built with _FORTIFY_SOURCE.

One way to make this more standard (and hence safer) is to, at every
malloc_usable_size() call, also 'reallocate' the block so that the
compiler can see the larger size.  This is done through a dummy
reallocator whose only purpose is to tell the compiler about the larger
usable size, it doesn't do any actual reallocation.

Florian Weimer pointed out that this doesn't solve the problem of an
allocator potentially growing usable size at will, which will break the
implicit assumption in systemd use that the value returned remains
constant as long as the object is valid.  The safest way to fix that is
for systemd to step away from using malloc_usable_size() like this.

Resolves systemd#22801.

(cherry picked from commit 7929e18)
thom311 added a commit to NetworkManager/NetworkManager that referenced this issue Feb 8, 2023
The idea of nm_free_secret() is to clear the secrets from memory. That
surely is some layer of extra snake oil, because we tend to pass secrets
via D-Bus, where the memory gets passed down to (D-Bus) libraries which
have no idea to keep it private. Still...

But turns out, malloc_usable_size() might not actually be usable for
this. Read the discussion at [1].

Stop using malloc_usable_size(), which seems unfortunate.

There is probably no secret relevant data after the NUL byte anyway,
because we tend to create such strings once, and don't rewrite/truncate
them afterwards (which would leave secrets behind as garbage).

Note that systemd's erase_and_free() still uses malloc_usable_size()
([2]) but the macro foo to get that right is terrifying ([3]).

[1] systemd/systemd#22801 (comment)
[2] https://github.com/systemd/systemd/blob/11c0f0659ecd82572c2dc83f3b34493a36dcd954/src/basic/memory-util.h#L101
[3] systemd/systemd@7929e18

Fixes: d63cd26 ('shared: improve nm_free_secret() to clear entire memory buffer')
thom311 added a commit to NetworkManager/NetworkManager that referenced this issue Feb 9, 2023
The idea of nm_free_secret() is to clear the secrets from memory. That
surely is some layer of extra snake oil, because we tend to pass secrets
via D-Bus, where the memory gets passed down to (D-Bus) libraries which
have no idea to keep it private. Still...

But turns out, malloc_usable_size() might not actually be usable for
this. Read the discussion at [1].

Stop using malloc_usable_size(), which seems unfortunate.

There is probably no secret relevant data after the NUL byte anyway,
because we tend to create such strings once, and don't rewrite/truncate
them afterwards (which would leave secrets behind as garbage).

Note that systemd's erase_and_free() still uses malloc_usable_size()
([2]) but the macro foo to get that right is terrifying ([3]).

[1] systemd/systemd#22801 (comment)
[2] https://github.com/systemd/systemd/blob/11c0f0659ecd82572c2dc83f3b34493a36dcd954/src/basic/memory-util.h#L101
[3] systemd/systemd@7929e18

Fixes: d63cd26 ('shared: improve nm_free_secret() to clear entire memory buffer')
(cherry picked from commit 8b66865)
lkundrak pushed a commit to NetworkManager/NetworkManager that referenced this issue Feb 9, 2023
The idea of nm_free_secret() is to clear the secrets from memory. That
surely is some layer of extra snake oil, because we tend to pass secrets
via D-Bus, where the memory gets passed down to (D-Bus) libraries which
have no idea to keep it private. Still...

But turns out, malloc_usable_size() might not actually be usable for
this. Read the discussion at [1].

Stop using malloc_usable_size(), which seems unfortunate.

There is probably no secret relevant data after the NUL byte anyway,
because we tend to create such strings once, and don't rewrite/truncate
them afterwards (which would leave secrets behind as garbage).

Note that systemd's erase_and_free() still uses malloc_usable_size()
([2]) but the macro foo to get that right is terrifying ([3]).

[1] systemd/systemd#22801 (comment)
[2] https://github.com/systemd/systemd/blob/11c0f0659ecd82572c2dc83f3b34493a36dcd954/src/basic/memory-util.h#L101
[3] systemd/systemd@7929e18

Fixes: d63cd26 ('shared: improve nm_free_secret() to clear entire memory buffer')
(cherry picked from commit 8b66865)
(cherry picked from commit 6e7fb78)
d-hatayama pushed a commit to d-hatayama/systemd that referenced this issue Feb 15, 2023
systemd uses malloc_usable_size() everywhere to use memory blocks
obtained through malloc, but that is abuse since the
malloc_usable_size() interface isn't meant for this kind of use, it is
for diagnostics only.  This is also why systemd behaviour is flaky when
built with _FORTIFY_SOURCE.

One way to make this more standard (and hence safer) is to, at every
malloc_usable_size() call, also 'reallocate' the block so that the
compiler can see the larger size.  This is done through a dummy
reallocator whose only purpose is to tell the compiler about the larger
usable size, it doesn't do any actual reallocation.

Florian Weimer pointed out that this doesn't solve the problem of an
allocator potentially growing usable size at will, which will break the
implicit assumption in systemd use that the value returned remains
constant as long as the object is valid.  The safest way to fix that is
for systemd to step away from using malloc_usable_size() like this.

Resolves systemd#22801.
amir73il pushed a commit to amir73il/man-pages that referenced this issue Apr 9, 2023
It might very well return a value larger than the actual usable size, so
writing to the excess bytes is Undefined Behavior.  There's absolutely
no promise about the value, except that it is no less than the size
that was once passed to malloc(3).

Link: <systemd/systemd#22801 (comment)>
Link: <https://inbox.sourceware.org/libc-alpha/20221124213258.305192-1-siddhesh@gotplt.org/T/>
Reported-by: Mingye Wang <arthur200126@gmail.com>
Reported-by: Siddhesh Poyarekar <siddhesh@gotplt.org>
Cc: DJ Delorie <dj@redhat.com>
Cc: Sam James <sam@gentoo.org>
Cc: Florian Weimer <fweimer@redhat.com>
Cc: Andreas Schwab <schwab@linux-m68k.org>
Cc: Zack Weinberg <zack@owlfolio.org>
Cc: Wilco Dijkstra <Wilco.Dijkstra@arm.com>
Signed-off-by: Alejandro Colomar <alx@kernel.org>
valentindavid pushed a commit to valentindavid/systemd that referenced this issue Aug 8, 2023
systemd uses malloc_usable_size() everywhere to use memory blocks
obtained through malloc, but that is abuse since the
malloc_usable_size() interface isn't meant for this kind of use, it is
for diagnostics only.  This is also why systemd behaviour is flaky when
built with _FORTIFY_SOURCE.

One way to make this more standard (and hence safer) is to, at every
malloc_usable_size() call, also 'reallocate' the block so that the
compiler can see the larger size.  This is done through a dummy
reallocator whose only purpose is to tell the compiler about the larger
usable size, it doesn't do any actual reallocation.

Florian Weimer pointed out that this doesn't solve the problem of an
allocator potentially growing usable size at will, which will break the
implicit assumption in systemd use that the value returned remains
constant as long as the object is valid.  The safest way to fix that is
for systemd to step away from using malloc_usable_size() like this.

Resolves systemd#22801.

(cherry picked from commit 7929e18)
(cherry picked from commit 34b9edd)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
7 participants