Add portable access to OS virtual memory manager #653

DmitryOlshansky · 2013-10-31T13:20:25Z

Preliminary pull for portable virtual memory management.

This needs a review and a nod from auto-tester.

UPDATE: and I was too optimistic with POSIX as standard in this area. Apparently I need FreeBSD and OSX "recipes" as well.

JakobOvrum · 2013-10-31T13:51:41Z

src/core/vmm.d

+    target OS virtual memory manager subsystem.
+
+    $(P 
+        The primary goal is not to provide full potential of every OS 


The newline in the source code makes it harder to spot, but there is a run-on sentence here. It should be split into two, or joined with a comma and a conjunction (perhaps "as").

JakobOvrum · 2013-10-31T14:08:44Z

If a bad argument to protectionFlags is indeed a logic error, it looks like most or all of these functions should be nothrow.

DmitryOlshansky · 2013-10-31T16:50:04Z

Thanks for proof-reading, that was a hasty write-up. I'll make adjustments.

Hm and I know why it passes the tests ... as I haven't included it in druntime makefiles at all.

JakobOvrum · 2013-10-31T17:31:11Z

src/core/vmm.d

+    }
+
+    /**
+        Reserve a $(D size) bytes of virtual memory rounded up 


+"block of"

JakobOvrum · 2013-10-31T17:42:31Z

Another grammar nitpick, but it should use the imperative mood of the verbs in all opening DDoc lines, which is currently hit and miss. The ones I spotted using the indicative mood are "commits", "decommits", "sets", and "undoes", which should be "commit", "decommit", "set" and "undo", respectively. Maybe this is taking it too far, but as it's a pet-peeve of mine I couldn't help but point it out...

edit:

Correction: in all opening DDoc lines for functions.

DmitryOlshansky · 2013-10-31T17:49:32Z

Yeah, you are quite a corrector :)

I think it's time to open this discussion somewhere on NG.

EDIT: I meant to continue this discussion...

MartinNowak · 2013-11-01T13:26:34Z

src/core/vmm.d

+        To have fine grained control over reserve and commit phases 
+        use $(LREF reserve) and $(LREF commit).
+    */
+    static void[] allocate(void* base, size_t size,


I'd prefer the names map/unmap instead of allocate/free. They are less confusable wih heap allocations.

map/unmap is good idea but then reserve/commit/decommit are getting somewhat out of loop.
So you reserve memory and then umap it.... Hm, sounds strange but might be OK.

MartinNowak · 2013-11-01T13:38:45Z

Another grammar nitpick, but it should use the imperative mood of the verbs in all opening DDoc lines

This is a good point.

So far only Win32 stuff is tested

Remove VMM struct. Few attempts to fix posix version. Add missing constants to sys.posix.unistd. Some tweaks to DDocs OS page size --> platform's page size. Linux and Windows now are forced to discard pages on decommit, unlike POSIX madvise which is overly lax on guarantees.

DmitryOlshansky · 2013-11-02T12:20:01Z

I've made another pass encorporating your suggestions and tried to figure out what the heck's wrong with our POSIX headers.

Anyone knows why only linux has plain (and good) madvise and others are featurinng only posix_madvise ?

MartinNowak · 2013-11-02T12:45:50Z

Anyone knows why only linux has plain (and good) madvise and others are featurinng only posix_madvise ?

As the name says posix_madvise is part of Posix, madvise is platform dependent. Therefor you find the latter only in coree.sys.linux.sys.mman.

DmitryOlshansky · 2013-11-02T13:59:11Z

Then I guess I then need to introduce core.sys.freebsd.sys.mman and core.sys.osx.sys.mman.
POSIX doesn't provide interesting semantics.

Aslo I'm considering an option is to provide 2 kinds of decommit:

Non-invalidating. Keep mapping valid but allow to reuse these pages on demand. The next access to that page may have the original conent or be a brand new zero-filled. Call it recycle or garbage as application indicates that it no longer needs the contents of a range.

Invalidating. Keep memory range but forcibly put pages to free list (or whatever). The next access to that page requires commit call, pages are brand new zero-filled. Application indicates that it no longer needs the commited memory.

Now in short I'm seeing that Linux & Windows do support invalidating version.
Windows, OSX and FreeBSD do support non-invalidating.
.

DmitryOlshansky · 2013-11-02T17:46:09Z

@dawgfoto Do you know what's the policy on system headers is - how one should go about creating them anyway?

For instance, when I look at C headers of BSD I see that there is a licence at top of it. I bet the same applies to Linux and/or whatever OS that is. Yet I haven't seen any notice of that sort in sys package (even Linux) - does that mean they are clean-room reconstruction of system headers? Can't we just do machine translation of each of corresponding C headers instead?

complexmath · 2013-11-07T01:07:19Z

Regarding system headers, there's a bit on it in the Boost license page (http://www.boost.org/users/license.html):

"Does the copyright and license cover interfaces too? The conceptual interface to a library isn't covered. The particular representation expressed in the header is covered, as is the documentation, examples, test programs, and all the other material that goes with the library. A different implementation is free to use the same logical interface, however. Interface issues have been fought out in court several times; ask a lawyer for details."

What I've been doing for the POSIX headers is looking at the OpenGroup spec, breaking what it says the header should contain into groups, and writing the D headers by hand using the actual system headers as a reference. So this approach doesn't use "the particular representation expressed in the header" which is what the Boost page says is the copyrightable part.

I don't know whether the output of a c2d translator would be considered infringing (though I'm inclined to think it would be), but that would be an unmaintainable mess if run on the BSD system headers anyway. Look at the definition of a POSIX struct in the BSD headers. I bet you have to look at 5 other headers not mentioned by the POSIX spec and wade through a bunch of different #ifdef cases dealing with various architectures and such. I absolutely do not want that in the Druntime headers if there's any reasonable way to avoid it, and so far I haven't seen an instance where it's necessary. My approach definitely means more work though, which is why the headers are created on demand... and preferably via a pull request from the person who wants the header in the first place, at least for the platforms she is requesting.

DmitryOlshansky · 2013-11-07T10:13:56Z

What I've been doing for the POSIX headers is looking at the OpenGroup spec, breaking what it says the header should contain into groups, and writing the D headers by hand using the actual system headers as a reference. So this approach doesn't use "the particular representation expressed in the header" which is what the Boost page says is the copyrightable part.

The idea is nice in that we'd have clean-room headers.
But recreateing by hand something that is constant motion (aside from POSIX standard) is fraught with error. Who would test all these APIs ? Why should we spent limitted manpower on something that OS maintainers already sweated out?

I don't know whether the output of a c2d translator would be considered infringing (though I'm inclined to think it would be), but that would be an unmaintainable mess if run on the BSD system headers anyway. Look at the definition of a POSIX struct in the BSD headers

The problem is that this mess has some point behind it. Also take a look at linux.sys.mman - it's already a mess and who would guarantee it's:
a) correct (on all platforms esp when it comes time to port somewhere else)
b) won't get out of sync tomorrow (with the new version of OS)
c) isn't severly behind the state of the art, as of 10-20 people who need it only 1 might do the pull, other just go away

As binary compatibility is of no concern to Linux folks, they may as well change a couple of constants here and there as long as source-level compatibility is intact.

This puts us in a silly position where basically C/C++ is so much better - they have fresh and correct headers.
Can't we just redistribute "System headers" with EXACTLY the same strings attached? People are already accepting this license when writing stuff for this OS in C/C++ anyway. Am I missing something big here?

DmitryOlshansky · 2013-11-08T21:35:41Z

I'm temporarily closing this to get some field testing of how the resulting code behaves on various platforms.
Will run some tests/benches and report back.

MartinNowak · 2013-11-09T05:45:08Z

@dawgfoto Do you know what's the policy on system headers is - how one should go about creating them anyway?

For posix start with the opengroup spec and fill in the values for each platform.
For OS dependent headers start by publically importing the posix header and then translate the C headers.
For linux the architecture dependent #include bits are currently declared inline.

Am I missing something big here?

Machine translation isn't trivial. I mostly use regex-replace when doing this by hand, but writing a generic tool is a lot of effort.

mleise · 2014-05-06T06:08:01Z

Exactly what we need!

I think getpagesize() is deprecated at least on Linux.
You don't need ; after }.
It is nice that you return all of the allocated memory even if the requested size was rounded up.
The little round up/down helpers are good.
You could go without huge pages in the first iteration if they cause too much trouble.
I haven't thought of the lock primitive yet, mainly because I didn't need it yet. There is one other primitive though: remapping. It basically changes what physical memory a virtual page points to. You can use it to look at different parts of a file or to write awesome circular buffers by mirroring the physical memory at the start to the end of the buffer.

mleise · 2014-05-06T06:31:05Z

Might be of interest: https://bugs.webkit.org/show_bug.cgi?id=65766#c17

DmitryOlshansky · 2014-05-06T20:21:44Z

Remapping frustratingly sucks on Win32, Basically it has to use FileMapping API then and carry around an explicit handle for the mapping along with the pointer...

mleise · 2014-05-06T23:55:44Z

Then leave that out for now, if it is such a can of worms.

JakobOvrum reviewed Oct 31, 2013
View reviewed changes

src/core/vmm.d

}

/**

Reserve a $(D size) bytes of virtual memory rounded up

Copy link

JakobOvrum Oct 31, 2013

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+"block of"

MartinNowak reviewed Nov 1, 2013
View reviewed changes

DmitryOlshansky added 5 commits November 1, 2013 19:58

add VMM functions, prepare some DDocs

a04c547

So far only Win32 stuff is tested

add vmm.d to makefiles

ae35ff1

Tweak DDocs as suggested by the review

1c0f255

reintroduce Posix version

585f96f

add more constants from BSD headers

512fe0e

DmitryOlshansky added 3 commits November 4, 2013 14:30

Forget sysconf, use unistd.h getpagesize call on *NIX

b0cc6c0

remove SC constants, no longer need these

7c3f138

restore ANON for other posix

71d41f5

DmitryOlshansky closed this Nov 8, 2013

DmitryOlshansky mentioned this pull request Apr 23, 2016

Add Virtual memory manager module #1549

Closed

Uh oh!

Add portable access to OS virtual memory manager #653

Add portable access to OS virtual memory manager #653

Uh oh!

Conversation

DmitryOlshansky commented Oct 31, 2013

Uh oh!

JakobOvrum Oct 31, 2013

Choose a reason for hiding this comment

Uh oh!

JakobOvrum commented Oct 31, 2013

Uh oh!

DmitryOlshansky commented Oct 31, 2013

Uh oh!

JakobOvrum Oct 31, 2013

Choose a reason for hiding this comment

Uh oh!

JakobOvrum commented Oct 31, 2013

Uh oh!

DmitryOlshansky commented Oct 31, 2013

Uh oh!

MartinNowak Nov 1, 2013

Choose a reason for hiding this comment

Uh oh!

DmitryOlshansky Nov 1, 2013

Choose a reason for hiding this comment

Uh oh!

MartinNowak commented Nov 1, 2013

Uh oh!

DmitryOlshansky commented Nov 2, 2013

Uh oh!

MartinNowak commented Nov 2, 2013

Uh oh!

DmitryOlshansky commented Nov 2, 2013

Uh oh!

DmitryOlshansky commented Nov 2, 2013

Uh oh!

complexmath commented Nov 7, 2013

Uh oh!

DmitryOlshansky commented Nov 7, 2013

Uh oh!

DmitryOlshansky commented Nov 8, 2013

Uh oh!

MartinNowak commented Nov 9, 2013

Uh oh!

mleise commented May 6, 2014

Uh oh!

mleise commented May 6, 2014

Uh oh!

DmitryOlshansky commented May 6, 2014

Uh oh!

mleise commented May 6, 2014

Uh oh!

Uh oh!