[RFC] Perhaps a shim is not necessary? #1
Comments
|
I rewrote the logic in C so it'll hopefully be more clear https://github.com/zhuyifei1999/zapps-poc/blob/master/zapps-crt0.c |
|
Awesome. |
|
I agree the shim process part of Zapps is potentially extraneous, and a somewhat unattractive hack. Much of what we've shipped and called Zapps so far is using a very placeholder, minimum-viable-product implementation of the shim, as well. It's big, bulky, and certainly can be improved -- even within the category of shims that are doing multiple exec's. And this goes somewhere even further than the multiple-exec space. Super awesome. I had some idea that something like this should be possible, but it's completely spectacular to see it fully worked, and even proven by working demo. |
|
So I guess most questions that occur to me will be about what kinda of interfaces/ABI/behavioral-contracts this becomes sensitive to, and what that will imply about portability/maintainability/fragility. A few questions and scattered first impressions, in no particular order:
I look forward to looking at this in greater depth soon! |
I think a lot of the functions are of two kinds - string functions and syscall functions.
Do you mean
And we need to patch auxv to provide the
It's basically the same code heh, slightly inferior. The commit message (zhuyifei1999/zapps-poc@9162056) describes what changed in functionality in the rewrite.
I don't see why it would cause an issue. That patch to That said, I haven't tested with musl or other libcs yet so I'm not 100% sure.
In the C version I used a pwrite on /proc/self/mem instead, which seems like a much better way to do this (https://offlinemark.com/2021/05/12/an-obscure-quirk-of-proc/). No more RWX ;) |
|
Wait I misread
No I didn't do this. The patch was to revert PT_INTERP back from PT_ZAPPS_INTERP. This doesn't modify any glibc code, but only the segmen headers of the ELF we are loading. The X bit was set because certain older gccs, .text starts within the first mapped page. I use the |
I tested musl with a cross-compiler. Interestingly in musl ld.so symlinks to libc.so: And it defines And it totally just works: As a control group comparison, this is invoking the ld.so directly on the normal binary: And this is if musl ld.so is loaded by kernel as the interpreter: So yeah, I think this method is pretty libc-agnostic :) other than the file name of the ld.so needs to be changed. |
|
I also took a look at musl source code to see if the path of ld.so is a concern (and musl source is much easier to read than glibc), since earlier I said
|
|
Do you have any thoughts on what the next steps would be, and how we should integrate this? I'm thinking we may want to maintain both the older boring'er way for a while, and try this in parallel. (I'm very conservative in some regards.) I don't have strong opinions about how we implement this, but if you have ideas I'm receptive. By the way, I see you published your new code under an MIT license, and I'm happy for any open-source license like that. But just as a heads up, we tend to do Apache2-OR-MIT for other stuff, including our existing code in this repo. (This is largely guided by the licensing policies of Protocol Labs, which affects several of us by default, and is also very open-source oriented and generally pretty pleasing.) I don't think it's any problem to have code that's MIT-only floating around, but if it's okay by you to also make it Apache2-OR-MIT, it might create moderately less kerfuffle if we decide to combine these works all into one repo. |
Sure |
|
Done. zhuyifei1999/zapps-poc@3fdfc38
I'm not familiar with how warptools does the building & packaging, so I don't think I'm the right person to ask about this. What I did was more of just a proof that the extra exec need not exist and what procedures are necessary to get rid of it. That said, I don't see it as any more complex than building a zapps-crt0.o in some known location and for most projects add an LDFLAGS to use it. |
|
I have another dumb question that has come to my mind :) With this shimless crt0 approach: if we wanted to add in some further preamble code to manipulate the environment variables the process sees, what options do we have? Will it be possible to do so with consistency? We haven't done this in the ldshim repo as it stands yet either, to be clear. But it's looking like something that might slip into our scope in the future. For context on why: tl;dr: it seems practical. (Perhaps a bit kludgey, also. But practical.) It seems that, in addition to our linker tricks to make the initial library loading work correctly and path-agnostically, there's a good number of programs out in the wild that that will also require some additional kicking in the shins to behave path-agnostically at runtime. And sometimes environment variables we set at launch are going to be the easiest (read: not patching at build time) way to do this. (We've started to see a few of these already. Also, Joey Hess (who has apparently been barking up the same tree as zapps for quite a while!) pointed out to us that in his experience working in a similar direction, there were lots of environment variables he found as the most sensible way to request fully path-agnostic behaviors at runtime. Examples he mentioned included LOCPATH, GCONV_PATH, GIT_TEMPLATE_DIR, MANPATH... but of course the individual cases aren't the point; more that it seems to come up fairly often in practice.) (I'll also say: I still don't love environment variables as "the" way to solve these problems -- the inheritability of env vars in particular just doesn't feel correct or clean at all in these scenarios. But it's looking like something we should have a place for in our bag of tricks.) |
I mean, the variables are on stack, and the stack has a very well defined layout (https://lwn.net/Articles/631631/). You can do whatever you want to, adding env vars, modifying them, or deleting them, before passing the control over to the dynamic loader. To be honest I also think env variables are wrong. If bash is a zapp, and someone uses it to call a non-zapp program the non-zapp program should not inherit bash's workarounds. Similarly if strace is a zapp and someone passes a non-zapp for strace to invoke, that non-zapp shouldn't inherit strace zapp's workarounds. |
Uh oh!
There was an error while loading. Please reload this page.
I was reading https://zapps.app/technology/ yesterday and it ocurred to me that the shim process seemed extraneous.
ld.sois a ELF dynamic object, so is a dynamically linked executable:Both are able to run, because there's an entry point:
But
ld.sodoes not have an INTERP ELF segment:This means, to start the execution of a dynamically linked object, the INTERP segment is optional as far as the kernel is concerned. If the object has no INTERP, the kernel will load it at a randomized address and jump to the entry point.
... which got me thinking, if the entry point is reachable, we can do in userspace the what used to be the kernel's job of loading
ld.sointo the address space, right? All we need to do is fix up the auxiliary vector to makeld.sobelieve nothing is out of the ordinary.So here is the POC:
I chose to write it in assembly because I was too lazy to mess with compiler options. I just want something that will work regardless of compiler. Rewriting most of it in C is on my TODO. Only the initial stage of the entry point and then the jump to
ld.sohave to be assembly, but the rest should be convertible to C.It also turned out that libc will require an INTERP segment when it's called (otherwise this assertion will occur: https://elixir.bootlin.com/glibc/glibc-2.36.9000/source/elf/rtld.c#L1291) so I patched it in at runtime. This unforunately meant that I have that page as RWX. I can re-mprotect it with the initial permissions but it needs a bit more code to find the right segment for the right permission bits.
Why do we care? I think one of the potential use cases where a jumploader could fail is in the case of binfmt-misc, or even setuids (though I'm not sure about the security of my approach either yet). In the case of binfmt-misc, the kernel would pass information about what's being executed to the binary via auxiliary vector in
Omode. This would be lost upon a re-exec. And for setuids, invokingld.sodirectly breaks setuid executable (though I'm not sure if this use case is something to support). And besides, saving an exec sounds cool since exec is very expensive process.Anyways, here's a demo of the POC:
The
absoluteonly have minimal rpath just to show what the output looks like for a normal executable:It cannot be relocated:
And this is the relocatable:
... which is relocatable like other zapps:
Wdyt?
The text was updated successfully, but these errors were encountered: