-
Notifications
You must be signed in to change notification settings - Fork 4.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Proposal: Expose POSIX functions #19958
Comments
Appendix A: Alternate namesAlternate Library/Namespace names :
Alternate class names :
An option we have when introducing Linux-specific functions is to either put them in the same class as the POSIX functions, or group them in to a new class: LinuxFunctions . |
If they are only POSIX, then wouldn't a variant on Would they intertop with Windows or Windows Subsystem for Linux if available? Or throw PlatformNotSupported? |
@eerhardt I assume you've looked at Mono.Posix - is there any value in reusing their API surface? I figure we can't reuse the package, as it calls into their runtime. |
@danmosemsft wrote:
|
Only the initial phase of added APIs are in the POSIX standard (according to http://pubs.opengroup.org/onlinepubs/9699919799/nframe.html). I plan on adding Linux-specific APIs as well in the future, as the need arises.
I don't think it is a goal to support these APIs on Windows proper. They will throw PlatformNotSupported exceptions. |
Yes, I've heavily based this proposal off of Mono.Posix and plan to reuse designs/approaches from it as much as possible. As for reusing their API surface directly:
|
One part that is missing (IMO) is what our rules for translating types would be. A |
This is a good point. Here is my first "stab" at it, and once we solidify on an approach, I'll add it to the proposal. For the most part, we will follow the same marshaling strategies employed internally in our existing
In the example above, I was expecting the Another thing to note is that while we should have guiding rules and principles, each API will need to make appropriate trade-offs on a case-by-case basis. We should be as consistent across the board as possible, but also allow flexibility for each API to do what is the most correct thing for it. |
Regarding Consider open(2):
Q: In what encoding is A: "What's an encoding?" Which is to say, the encoding is wholly unspecified, and it's not difficult to create a filesystem entry which may contain invalid UTF-8:
...and I do mean "may". On macOS Sierra, the above creates a file named Update: What's Linux do? For a co-worker of mine in a Japanese locale,
i.e. the filename is a question mark In short, it isn't necessarily sane to assume that filenames retrieved from e.g. readdir(3) are in any known format, and this is a problem doubly faced by Mono's System.IO largely ignores this problem, and just assumes that user space is UTF-8, as is good and proper. Mono.Unix has a I'm less certain about what CoreFX should do about this. It's really tempting to just "require" that user space be UTF-8 -- though that also introduces questions around normalization -- and otherwise ignore the problem. Alternatively, it would be Really Handy™ if System.IO "somehow" handled non-UTF-8 filenames, at which point the POSIX functions could follow suit. |
Various replies to an earlier comment:
While getgrnam(3) returns a pointer into static memory, managed use of So I'd argue it's a possibly-thread-safe method. Which doesn't necessarily help anything...
This makes lots of sense, limiting project scope. That said, considering them as a part of a future project should be done now, to attempt to help with naming and organization. In the original work for Mono.Posix and the "version 2" Mono.Unix, everything was placed into the same namespace, both high-level wrappers and low-level types. This was okay for Mono.Posix, as there are only 14 types in that namespace, but During development of Mono.Unix, lots of additional methods were bound, which required binding a number of additional types. It was during this process that the Thus, whether or not high-level types exist is of interest and concern in Phase 1, even if they won't be added until later (if ever). Perhaps the appropriate fix is to "invert" what Mono.Unix does, and have low-level types in a Whatever is done, we should keep the larger picture in mind.
I'm not quite sure I understand the logic here. File descriptors are "Convention" aside, such a choice/requirement means that close(2) can't be "just" a P/Invoke into
(The above is certainly true for
...what?! I do not understand why Additionally, |
.NET Core's System.IO also ignores this problem. On the managed side, the DllImport uses |
My opinion is
100% agreed. My current thinking is these low-level, native APIs go in
See https://github.com/dotnet/corefx/issues/2904 and dotnet/corefx#4791. Basically it assists in SafeHandle usages. If we used
Yes, in .NET Core all of our interop with native Unix functions goes through a C intermediary, for example
Oops, this was a mistake to make a blanket statement. I had quickly looked at a couple of our internal interop calls and assumed that was the rule we used when first creating the shims. The rules we used are located at https://github.com/dotnet/corefx/blob/master/Documentation/coding-guidelines/interop-guidelines.md#unix-shims, which say "it is most convenient to line up with the managed int as int32_t (e.g. scratch buffer size for read/write), but sometimes we need to handle huge sizes (e.g. memory mapped files) and therefore use uint64_t.". Maybe to be as consistent as possible across these APIs, we should just always map
I really struggle on what to do here because IMO the boat has sailed in .NET for using signed values for lengths/indexes into collections. If these APIs use an unsigned integer, callers will always need to cast when passing in Thoughts? Opinions? Should these be unsigned or signed? |
The only reason that comes to mind, and the reason I bound
Mono.Unix.Native has been using |
The problem with ...or go with the Mono.Unix approach of a custom marshaler, but that certainly adds additional runtime overhead. It Works™, and allows you to have an easier-to-use "Better", arguably, would be to implement a solution that There's also the question of whether it's a scenario worth worrying about. |
There is no reason to create a new API when a perfectly working, debugged, battle-tested version exists already in the form of Mono.Posix. There are no good reasons to not use the existing Mono.Posix:
Not worth creating a different API for that reason.
It does not hurt anyone. Nothing prevents us from using
Not a reason to not bring it. A year from now, you will realize "Shoot, those would be useful, guess we should bind them". You talk about this being "Phase 1". The problem is that you do not have a roadmap or understanding for what "Phase N" will be. So we can dismiss this.
This is because you have a narrow view of the problem, summarized in this incremental approach at designing the API that you call "Phase 1". You are trying to bind APIs blindly and you think that this will get the job done. But it won't, it is only the start of a process. Once the API is out there, you will get real world feedback, you will come to the same conclusions that we did - except that you would have created a fork in the road, and slowed down everyone in the process by delivering your currently sub-standard API instead of embracing the battle tested implementation. Higher-level convenience APIs that are there because they addressed real-world use cases and reduced user bugs. Use cases that exist from almost 15 years of building Unix applications with .NET, a space that you are only entering now. You are not forced to use them if you do not want them.
This is good guidance for newcomers and people that are not familiar with the domain. Let me explain why this is not a problem. Let us start with the basics. This is an implementation detail, so if there is a problem with this approach, it can trivially be changed without affecting the API surface. The core issue with P/Invoke into libc is whether you can express the ABI requirements of calling a function. This happens in particular with functions like On the other hand we have APIs like If the day comes where we find a platform where P/Invoking a specific method of libc does not work, we would change the implementation and it would not affect the API surface, nor 99.9% of the existing implementation. We are deeply familiar with this problem - after all, it was my team that provided the guidance to the .NET Core team on why they should not P/Invoke into libc directly for a series of APIs on the early days of the port. Work that had to be done later on, as the team discovered the very problem that we had described to them. |
@migueldeicaza - you have me convinced. I agree that we should port I took a first stab at building it for .NET Core here: https://github.com/eerhardt/mono/commits/MonoPosixNETCore. I've run a few simple functions on .NET Core 2.0 on my mac, which worked. I also hit a bug in .NET Core 2.0: https://github.com/dotnet/coreclr/issues/9287, which needs to be fixed. Overall, I think an approach we can take would work like this:
This NuGet package would be in the same category as the All the APIs of Mono.Posix that work on .NET Core 2.0 would be available. If we agree on this overall approach, I can close out this issue and open a new issue on the mono repo to start the work of building this new NuGet package. |
Hello @eerhardt This is music to my ears. I do like the approach. One thing that we have done in the past for some pieces of Mono code is to extract them into a separate top-level module (and we have a script that extracts the code with the complete Git history) and then we have taken a dependency from Mono into the module, and .NET core takes a dependency into that module as well. We can start with your suggestion today, but we can also extract the code (it will take us just a couple of days to get it done). |
I would be fascinated to learn how this works. I have been unable to achieve this with NuGet v3+. NuGet/Home#3970 |
@jnm2 - The classic example is the You would just also add in the managed IL assembly into the same package. I'm unsure on a "clean and easy" way of enabling this using MSBuild or the
And just replace the |
I'm relieved that I finally have something that works! .nuspec is perfect. I have some complaints. 1. DocsI can't find the magic strings 2. Concerning NuGet v2 (net462 .csproj + packages.config)NuGet v2 (packages.config) projects will not copy the native asset to the build output at all. Not a problem because everyone has moved to NuGet v3 (.csproj + project.json) and soon v4, but I want a way to fail the package installation for NuGet v2 projects. 3. Concerning NuGet v3 (net462 .csproj + project.json)(To clarify, when I say project.json, I'm not talking about .NET Core. I'm talking about a traditional net462 .csproj + project.json.) Ideally I'd use
This means that the same file is duplicated three times in the nupkg just in order to show up in the build output of .NET apps of each of the three types. I don't like the duplication, but mostly what I don't like is the fact that there's no way to constrain it to I would expect something hierarchical- anything in 4. Concerning NuGet v4 (net462 .csproj, CPS msbuild)If I use the Now the nice thing is that unlike NuGet v3, NuGet v4 seems to let me skip adding the This means, in order to work properly with NuGet v4 projects, I must package the same DLL seven times at least, and I'm still probably missing something important:
Something seems broken here. In each of these scenarios, it seems that either I'm missing crucial documentation, or the design has not really been thought through yet. (If the design is still in progress, I'd appreciate knowing which issues to subscribe to.) So I'm happy that I can hack together a nuspec that works. Is there any way to ease the pain? |
There are some "runtimes" sections here: https://docs.microsoft.com/en-us/nuget/create-packages/project-json-and-uwp that describe this feature when it was added in NuGet v3. Another good doc on RIDs is here: https://github.com/dotnet/corefx/blob/master/pkg/Microsoft.NETCore.Platforms/readme.md These docs should help solve your problems for NuGet v4. There is a "fallback" strategy as outlined in the "How does NuGet use RIDs?" section of the 2nd document. Feel free to log issues in https://github.com/nuget/Home for things that don't work. |
I am wondering if this API shouldn't be cast in stone yet. There is discussion going on around native int And other native data types and if that goes through should this API use those types rather than fixing in cases like
|
The current plan is to package Mono.Posix into a NuGet package that is compatible with .NET Core 2.0. This would involve using the existing APIs exposed by this library. My understanding is @bholmes is working on this. @bholmes - any update? |
Yes there is some progress. There is a pull request in Mono that has the changes to Mono.Posix that are needed. Next I have a Mono.Posix folder in a fork that will handle build and test scripts for .Net Core 2.0. Finally here is the Jenkins location where the package is built. You can grab a nupkg there if you want to test. |
I'm going to close out this issue, since this proposal is no longer relevant. We now have a version of Mono.Posix that works on .NET Core 2.0 published to https://www.nuget.org/packages/Mono.Posix.NETStandard/1.0.0-beta1. Anyone who is interested, please try it out and give feedback. Thanks, @bholmes for all the work you and your team did to making it easier to call these common Unix and Linux APIs. |
To allow developers to develop applications on Unix operating systems, .NET Core should contain a library exposing a set of common Unix and Linux APIs.
Rationale
Today, .NET Core programs running on Unix do not have the ability to make calls into operating system functions in a portable manor, without writing native code. There are Unix-specific functions/capabilities that are not exposed in our current set of APIs. For example,
chmod
andchown
are commonly used functions on Unix to set file permissions, but we have noSystem.IO
APIs that expose this functionality.It is hard to P/Invoke directly into “libc” from managed code because of type-size differences across platforms, and constants/error codes being different across platforms.
Design tenets
A guiding tenet is to keep any additional functionality and policy out of these exposed APIs. Instead, we will simply be exposing the OS function as raw as possible.
One place where we will be adding functionality is to make the .NET APIs work across our supported platforms as seamless as possible.
A naming tenet will be to use the same name and casing as the C API, this includes method, parameter, enum value, and field names. This allows people to consult the appropriate man page if necessary.
Example Usages
APIs to Expose
The exposed functions will be implemented in phases. To determine the first phase of functions, I have analyzed Mono, Go, and Phython system functions. Phase 1 of this work will include functions that are:
That results in the following list of functions for Phase 1:
The text was updated successfully, but these errors were encountered: