Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.
Sign upUse memfd_create and shm_open instead of mkstemp #140
Conversation
| @@ -982,6 +970,13 @@ unsafe fn create_shmem(name: *const c_char, length: usize) -> c_int { | |||
| fd | |||
| } | |||
|
|
|||
| #[cfg(target_os="android")] | |||
| unsafe fn create_shmem(name: *const c_char, length: usize) -> c_int { | |||
| let fd = ashmem_create_region(name, length); | |||
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 9, 2017
Author
Collaborator
NB: I have not tested the android code yet. This is more or less a placeholder for now.
This comment has been minimized.
This comment has been minimized.
| @@ -1033,6 +1058,11 @@ fn is_socket(fd: c_int) -> bool { | |||
|
|
|||
| // FFI stuff follows: | |||
|
|
|||
| #[cfg(target_os="linux")] | |||
| unsafe fn memfd_create(name: *const c_char, flags: usize) -> c_int { | |||
| syscall!(MEMFD_CREATE, name, flags) as c_int | |||
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 9, 2017
Author
Collaborator
Kind of less than ideal in my books, but since glibc doesn't provide an wrapper, this is the recommended method. Perhaps we should make shm_open the default and this enabled by a feature flag?
This comment has been minimized.
This comment has been minimized.
antrik
Feb 9, 2017
Contributor
That's odd... Do you know whether glibc just doesn't provide a wrapper yet, or it doesn't intend to? (And if so, why?)
I don't think it's a reason to disable it by default though. It certainly would be useful however to have a --force-posix-shm flag along the lines of --force-inprocess, so we can easily test the code locally and in CI.
This comment has been minimized.
This comment has been minimized.
| @@ -649,10 +649,69 @@ fn make_socket_lingering(sockfd: c_int) -> Result<(),UnixError> { | |||
| Ok(()) | |||
| } | |||
|
|
|||
| struct BackingStore { | |||
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 9, 2017
Author
Collaborator
In my original design, this played a much bigger role. The BackingStore doesn't really do much as is, and could easily be removed. We really rely on the create_shmem function.
This comment has been minimized.
This comment has been minimized.
antrik
Feb 9, 2017
Contributor
While it didn't end up being platform-specific -- which I suppose was the reason you introduced it -- I still think it's a useful abstraction. Dealing with raw FDs is always quite fragile; anything that isolates this is good.
|
@Manishearth the dedicated MacOS back-end already uses Mach memory objects. This is only about platforms using the Unix back-end. |
TIL, nice. |
|
This looks pretty good already :-) (And steals an item from my ToDo list -- except that I probably wouldn't have bothered adding anything but POSIX SHM myself...) My major qualm is that the changes are in a sub-optimal order: causing a lot of unnecessary code churn, and making review of individual commits very weird... It would be much more logical to introduce the generic POSIX SHM implementation first, completely replacing the temp file one in one clean atomic commit; and only later bringing in the more platform-specific variants. |
| } | ||
|
|
||
| impl BackingStore { | ||
| pub unsafe fn create(length: usize) -> BackingStore { |
This comment has been minimized.
This comment has been minimized.
antrik
Feb 9, 2017
Contributor
This method doesn't need to be unsafe -- there is nothing the caller can do to break stuff by invoking it. Only the implementation is unsafe.
(Admittedly, this was also true of the original stand-alone function... Maybe you could fix this first in a separate commit -- not crucial though...)
Also, since this is the primary constructor method, I believe convention demands it to be called new()...
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 10, 2017
Author
Collaborator
Good point. I renamed the function to new and moved the code inside of an unsafe block. I probably could have moved only parts of the code inside the unsafe block, but I thought this looked more readable. I'm okay with changing that though.
|
|
||
| pub fn from_fd(fd: c_int) -> BackingStore { | ||
| BackingStore { | ||
| fd: fd |
This comment has been minimized.
This comment has been minimized.
antrik
Feb 9, 2017
Contributor
Nitpick: always terminate the last line in a multi-line initialiser with , as well.
(I just noticed that I didn't catch the same thing in the mio PR...)
This comment has been minimized.
This comment has been minimized.
| @@ -649,10 +649,69 @@ fn make_socket_lingering(sockfd: c_int) -> Result<(),UnixError> { | |||
| Ok(()) | |||
| } | |||
|
|
|||
| struct BackingStore { | |||
This comment has been minimized.
This comment has been minimized.
antrik
Feb 9, 2017
Contributor
While it didn't end up being platform-specific -- which I suppose was the reason you introduced it -- I still think it's a useful abstraction. Dealing with raw FDs is always quite fragile; anything that isolates this is good.
| @@ -8,6 +8,7 @@ repository = "https://github.com/servo/ipc-channel" | |||
|
|
|||
| [features] | |||
| force-inprocess = [] | |||
| force-shm = [] | |||
This comment has been minimized.
This comment has been minimized.
antrik
Feb 9, 2017
Contributor
Probably best to add this in whatever commit first introduces the more specific variants.
|
|
||
| #[cfg(not(target_os="android"))] | ||
| const TEMP_FILE_TEMPLATE: &'static str = "/tmp/ipc-channel-shared-memory.XXXXXX"; | ||
| const TEMP_FILE_TEMPLATE: &'static str = "/ipc-channel-shared-memory."; |
This comment has been minimized.
This comment has been minimized.
| unsafe fn maybe_unlink(c: *const c_char) -> c_int { | ||
| libc::unlink(c) | ||
| #[cfg(any(feature="force-shm", target_os="freebsd"))] | ||
| #[inline] |
This comment has been minimized.
This comment has been minimized.
antrik
Feb 9, 2017
Contributor
Explicit inlining is useful only for functions invoked from other crates.
| let fd = create_shmem(string.as_ptr(), length); | ||
| BackingStore { | ||
| fd: fd | ||
| } |
This comment has been minimized.
This comment has been minimized.
antrik
Feb 9, 2017
Contributor
Could use Self::from_fd() I guess? Not sure whether that's indeed more readable though...
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 10, 2017
Author
Collaborator
Debated this for a while, but I went ahead and also updated this.
| 0 | ||
| unsafe fn create_filename() -> CString { | ||
| let mut tmp_string = TEMP_FILE_TEMPLATE.to_owned(); | ||
| let addition: String = thread_rng().gen_ascii_chars().take(6).collect(); |
This comment has been minimized.
This comment has been minimized.
antrik
Feb 9, 2017
Contributor
@asajeffrey spent some effort removing use of thread_rng() in Servo, since it uses up an FD for each thread...
There is actually no need for this to be random -- so I think a global lazy_static! atomic counter would do just fine. It would be less overhead, too.
(In fact I'm not sure there is any value in using unique names with memfd at all: the name is only for debugging here; and since we don't keep track of it after initial creation of the FD anyway, I don't think they really help telling the FDs apart...)
Seems more tricky for the POSIX SHM interface though, since the names are global there. Probably need to use the PID along with a counter and timestamp. (We need the timestamp part, since theoretically a process might die before releasing the SHM object, and later another process reusing the PID might collide with that...)
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 10, 2017
Author
Collaborator
I still have some work to do on this. In the meantime I removed the use of thread_rng() in this function and used a combination of the pid and timestamp. To be completely honest, I totally forgot to add the counter
Seems more tricky for the POSIX SHM interface though
Yes, AFAIK both of the other platforms should be fine with the lazy_static, but it gets a lot trickier with any platforms using the shm_open implementation.
| #[cfg(any(feature="force-shm", target_os="freebsd"))] | ||
| #[inline] | ||
| unsafe fn create_shmem(name: *const c_char, length: usize) -> c_int { | ||
| let fd = libc::shm_open(name, libc::O_CREAT | libc::O_RDWR, 0660); |
This comment has been minimized.
This comment has been minimized.
antrik
Feb 9, 2017
Contributor
Why 660, not 600?
(Though in fact it probably should be 000, since we don't ever open the same object again anyway -- we just clone the existing FD, which is already open with O_RDWR...)
Also, 0660 doesn't do what you think it does :-) In Rust, you'd need 0o660.
Last but not least, it's probably good to add O_EXCL just to be safe.
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 10, 2017
Author
Collaborator
Updated.
Also, 0660 doesn't do what you think it does :-) In Rust, you'd need 0o660.
Old habits die hard
| unsafe fn create_filename() -> CString { | ||
| let mut tmp_string = TEMP_FILE_TEMPLATE.to_owned(); | ||
| let addition: String = thread_rng().gen_ascii_chars().take(6).collect(); | ||
| tmp_string.push_str(&addition); |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
Just to be clear: it's especially the second commit that is weird, since it refactors code that is obsoleted piece by piece by later commits... It would be much cleaner to do any refactoring necessary on top of the POSIX SHM code as the new base-line, before adding the specific Linux and Android back-ends. |
Good feedback! Reordered the commits and condensed a few commits. |
|
The commit message of the third commit doesn't fit any more -- (BTW, using "Android platform" and "Linux platform" might be slightly misleading in this context, since these are just special cases of the |
| @@ -10,6 +10,7 @@ os: | |||
| env: | |||
| - FEATURES="unstable" | |||
| - FEATURES="unstable force-inprocess" | |||
| - FEATURES="unstable force-posix-shm" | |||
This comment has been minimized.
This comment has been minimized.
antrik
Feb 10, 2017
Contributor
I wonder whether there is a way to enable this only on the Linux builder? It makes no sense on MacOS...
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 11, 2017
Author
Collaborator
Yeah, I though that was a little strange... I'l look into this
| // and threading that value into Servo. | ||
| // https://code.google.com/p/android/issues/detail?id=19017 | ||
| 0 | ||
| fn create_string() -> CString { |
This comment has been minimized.
This comment has been minimized.
antrik
Feb 10, 2017
Contributor
create_string() doesn't really have a lot of meaning... Should be something like create_shm_name() I'd say -- though actually I'm not convinced it's useful to have a separate function for this at all...
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 11, 2017
Author
Collaborator
though actually I'm not convinced it's useful to have a separate function for this at all
Originally I was going to generate the name differently for different OSes, but in the end I thought it was probably best to keep it simple. I'll add the name creation to the BackingStore::create method.
| // https://code.google.com/p/android/issues/detail?id=19017 | ||
| 0 | ||
| fn create_string() -> CString { | ||
| let epoc = UNIX_EPOCH.elapsed().unwrap_or(Duration::new(0, 0)); |
This comment has been minimized.
This comment has been minimized.
antrik
Feb 10, 2017
Contributor
I think you can just unwrap() here -- time elapsed since epoch should always be valid.
(Weird that there seems to be no simpler way to obtain this... And also weird that there seems to be no way to obtain a global monotonic timestamp. Doesn't really matter though in this case, as with a precise timestamp along with the PID and a counter, a collision is well-nigh impossible even if system time does jump back...)
epoc is a weird variable name, BTW... Just timestamp I'd say?
This comment has been minimized.
This comment has been minimized.
| let epoc = UNIX_EPOCH.elapsed().unwrap_or(Duration::new(0, 0)); | ||
| let pid = unsafe { | ||
| libc::getpid() | ||
| }; |
This comment has been minimized.
This comment has been minimized.
antrik
Feb 10, 2017
Contributor
For such a simple unsafe expression, I believe it's more customary to put it on a single line.
| let pid = unsafe { | ||
| libc::getpid() | ||
| }; | ||
| CString::new(&*format!("/ipc-channel-shared-memory.{}.{}.{}", |
This comment has been minimized.
This comment has been minimized.
antrik
Feb 10, 2017
Contributor
You don't need the &* dance here: CString::new() actually wants an owned string (or some other byte vector) -- if you turn it into a slice first, this just means it will have to clone it unnecessarily to gain ownership...
| unsafe fn maybe_unlink(c: *const c_char) -> c_int { | ||
| libc::unlink(c) | ||
| #[cfg(any(feature="force-posix-shm", not(any(target_os="linux", target_os="android"))))] | ||
| unsafe fn create_shmem(name: *const c_char, length: usize) -> c_int { |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
| libc::unlink(c) | ||
| #[cfg(any(feature="force-posix-shm", not(any(target_os="linux", target_os="android"))))] | ||
| unsafe fn create_shmem(name: *const c_char, length: usize) -> c_int { | ||
| let fd = libc::shm_open(name, libc::O_CREAT | libc::O_RDWR | libc::O_EXCL, 0o600); |
This comment has been minimized.
This comment has been minimized.
antrik
Feb 10, 2017
Contributor
A quick check seems to confirm that this works with 0o000 as well, as I expected :-)
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 11, 2017
Author
Collaborator
I just tested this on FreeBSD and unfortunately you can't unlink if you don't open with at least 0o600
012bf63
to
3fd57cc
| env: FEATURES="unstable" | ||
| - os: osx | ||
| rust: nightly | ||
| env: FEATURES="unstable force-inprocess" |
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 11, 2017
Author
Collaborator
This is a bit more verbose, but it doesn't build with force-posix-shm on osx.
This comment has been minimized.
This comment has been minimized.
antrik
Feb 12, 2017
Contributor
Since this is a somewhat major change, I wonder whether it might be better to do it in a separate commit?
This comment has been minimized.
This comment has been minimized.
|
Updated to address comments from previous review. I also removed the OS specific commit for android. From some continued testing and code review, I realized that the OS specific code for android under the |
| let pid = unsafe { libc::getpid() }; | ||
| let name = CString::new(format!("/ipc-channel-shared-memory.{}.{}.{}", | ||
| count, pid, | ||
| timestamp.as_secs())).unwrap(); |
This comment has been minimized.
This comment has been minimized.
antrik
Feb 12, 2017
Contributor
Why did you drop the nanosecond part? It seems perfectly plausible for a process to exec() (thus resetting the counter while keeping the PID), and then create new shared memory objects all within a single second...
With a more precise timestamp on the other hand, the time it takes to exec() should be sufficient to avoid collisions.
(To be honest, it feels a bit fragile even with that, if running on a system with low timer resolution... I see no obvious way to fix this though, without re-introducing some sort of random element :-( )
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 13, 2017
Author
Collaborator
Why did you drop the nanosecond part? It seems perfectly plausible for a process to exec() (thus resetting the counter while keeping the PID), and then create new shared memory objects all within a single second...
I didn't think about the exec scenario. This is definitely the part of the PR that I'm most unsure about at the moment. I guess if we go back to the pid, counter, second, nanosecond combo I'd be happy with it, but I also started to wonder if some sort of random hash might be better...
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 13, 2017
Author
Collaborator
Nanosec part was added back in. So now we have /ipc-channel-shared-memory.<count>.<pid>.<seconds>.<nanoseconds>
This comment has been minimized.
This comment has been minimized.
antrik
Feb 14, 2017
Contributor
I'm really uncertain myself whether a random hash wouldn't be preferable. I don't know whether other methods of generating random numbers -- such as weak_rng(), or perhaps even StdRng::new() in a lazy_static -- would avoid the FD use problem. If so, it might be preferable, since it's probably a tick simpler all in all... Really not sure though. I don't see a strong motivation to change it at this point.
| env: FEATURES="unstable" | ||
| - os: osx | ||
| rust: nightly | ||
| env: FEATURES="unstable force-inprocess" |
This comment has been minimized.
This comment has been minimized.
antrik
Feb 12, 2017
Contributor
Since this is a somewhat major change, I wonder whether it might be better to do it in a separate commit?
| unsafe { | ||
| let fd = libc::shm_open(name.as_ptr(), | ||
| libc::O_CREAT | libc::O_RDWR | libc::O_EXCL, | ||
| 0o600); |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 13, 2017
Author
Collaborator
So the FreeBSD man page says shm_unlink needs write privileges, but testing is showing it needs read-write.
This comment has been minimized.
This comment has been minimized.
antrik
Feb 13, 2017
Contributor
Way to keep programmers on their toes ;-)
Probably good to add a comment for this?
This comment has been minimized.
This comment has been minimized.
|
|
||
| impl BackingStore { | ||
| pub fn new(length: usize) -> BackingStore { | ||
| let count = GLOBAL_SHM_COUNT.fetch_add(1, Ordering::SeqCst); |
This comment has been minimized.
This comment has been minimized.
antrik
Feb 12, 2017
Contributor
While the documentation on Ordering is unfortunately pretty useless to mere mortals, after doing some digging, I'm fairly confident that Relaxed would do here...
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 13, 2017
Author
Collaborator
Haha. Yeah, I always forget that Relaxed is the same as LLVM's definition of monotomic.
| @@ -825,7 +880,7 @@ fn recv(fd: c_int, blocking_mode: BlockingMode) | |||
| let bytes_read = try!(cmsg.recv(fd, blocking_mode)); | |||
| main_data_buffer.set_len(bytes_read - mem::size_of_val(&total_size)); | |||
|
|
|||
| let cmsg_fds = cmsg.cmsg_buffer.offset(1) as *const c_int; | |||
| let cmsg_fds = CMSG_DATA(cmsg.cmsg_buffer) as *const c_int; | |||
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 13, 2017
Author
Collaborator
I can submit this as a separate one... There are a few other things I've found that I'll want to fix too
This comment has been minimized.
This comment has been minimized.
3a3d6f6
to
cf33cec
| @@ -72,6 +68,8 @@ lazy_static! { | |||
| }; | |||
| } | |||
|
|
|||
| static GLOBAL_SHM_COUNT: AtomicUsize = ATOMIC_USIZE_INIT; | |||
This comment has been minimized.
This comment has been minimized.
antrik
Feb 14, 2017
Contributor
I totally forgot to comment on this: I don't think the GLOBAL_ prefix is useful. I haven't ever seen something like that used with Rust. (The all-caps already identify it clearly as a global.)
Maybe also good to add a comment that this is used for creating the unique IDs?
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 15, 2017
Author
Collaborator
Added a comment and removed GLOBAL_. I left the SHM_ prefix and named the lazy_static pid SHM_PID. I don't really like the SHM_ prefix, but I wanted to make it a little clearer what it was used for.
This comment has been minimized.
This comment has been minimized.
antrik
Feb 16, 2017
Contributor
Personally, I don't think PID should have a SHM_ prefix -- while its (currently) used only for this, the PID itself is a general property of the process, not in any way SHM-specific...
I don't feel strongly about it though, and I don't think this warrants another iteration :-)
| pub fn new(length: usize) -> BackingStore { | ||
| let count = GLOBAL_SHM_COUNT.fetch_add(1, Ordering::Relaxed); | ||
| let timestamp = UNIX_EPOCH.elapsed().unwrap(); | ||
| let pid = unsafe { libc::getpid() }; |
This comment has been minimized.
This comment has been minimized.
antrik
Feb 14, 2017
Contributor
I wonder whether it would be useful to put this in a lazy_static. I don't know whether this is cached somehow -- but if it isn't, and performs an actual system call every time, this could have a somewhat significant performance impact...
This comment has been minimized.
This comment has been minimized.
dlrobertson
Feb 15, 2017
Author
Collaborator
Sorry, you mentioned this earlier but I didn't get around to it. Updated per the above comment.
This comment has been minimized.
This comment has been minimized.
antrik
Feb 16, 2017
Contributor
Nah, I actually only mentioned using lazy_static for the counter before -- not realising that an atomic is the more obvious choice for that one...
Awesome! PR submitted and |
|
@antrik I should have brought this up sooner, but |
|
I don't think submitting "fake PRs" is a sustainable workflow. Unless we can include Servo testing in |
|
@danlrobertson Linux 3.17 is a pretty steep requirement indeed... Build-time feature detection is not sufficient, as the official nightlies for example need to be portable. We could attempt run-time detection -- but that would complicate the code, making it more error-prone and harder to test. I wonder whether it's really worthwhile under these conditions? Maybe it's best to make |
Agreed. Not having a solid definition of a "major" change is complicates things more (I'm also of the opinion that a "small" change is almost just as likely to introduce problems). IMO keeping
I'm starting to lean towards the |
|
Just |
Refactor the backing store functionality of OsIpcSharedMemory (create_memory_backing_store, map_file). Create a new structure BackingStore that contains the create and map_file functions which implement the same functionality.
Use POSIX shm_open/shm_unlink for unix OSes instead of mkstemp. In addition, move the platform specific code used to create the backing store out into a separate function create_shmem.
[WIP] test run on servo Ignore this PR. This is just a test run for servo/ipc-channel#140. <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/15632) <!-- Reviewable:end -->
|
Updated to include a |
|
The commit message for the last commit needs updating. (While at it, might be useful to mention why it required switching to the explicit build matrix? Probably quite surprising at first otherwise...) |
| libc::free(string_buffer as *mut c_void); | ||
| assert!(libc::ftruncate(fd, length as off_t) == 0); | ||
| fd | ||
| #[cfg(any(not(feature="memfd"), target_os="freebsd"))] |
This comment has been minimized.
This comment has been minimized.
antrik
Feb 21, 2017
Contributor
This should be written in terms of not(target_os="linux"). (It's the default implementation, used in all cases unless we have a more specific one...)
In fact it's probably clearer to write it as #[cfg(not(all(target_os="linux", feature="memfd")))] -- not least because that way it's obvious that it's complementary to the other one.
(I'd put the linux bit before the memfd one, since memfd is dependant on linux... Not really important though.)
[WIP] test run on servo Ignore this PR. This is just a test run for servo/ipc-channel#140. <!-- Reviewable:start --> --- This change is [<img src="https://reviewable.io/review_button.svg" height="34" align="absmiddle" alt="Reviewable"/>](https://reviewable.io/reviews/servo/servo/15632) <!-- Reviewable:end -->
For linux OSes allow the use of memfd_create for the adventurous souls that dare to do so. This feature should be enabled with the memfd features flag and should only be available on linux. Note: glibc does not currently provide a wrapper for this syscall yet. As a result, it is recommended to use `syscall` directly. See the man pages for details.
Update the travis CI config to include a build forcing the memfd feature to be used. Switching to an explicit build matrix was required as the memfd feature is not used for any target OSes other than linux.
Good catch.
Right, my bad... I didn't correctly migrate the |
|
@bors-servo r+ |
|
|
Use memfd_create and shm_open instead of mkstemp Remove the use of `mkstemp` which creates a file in `/tmp`. Instead use the following - `linux` - [`memfd_create`](http://man7.org/linux/man-pages/man2/memfd_create.2.html) - `freebsd` - [`shm_open`](https://www.freebsd.org/cgi/man.cgi?query=shm_open&sektion=2)
|
|
dlrobertson commentedFeb 9, 2017
•
edited
Remove the use of
mkstempwhich creates a file in/tmp. Instead use the followinglinux-memfd_createfreebsd-shm_open