New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.NET Core support and CoreCLR embedding - cross-platform API #96

Open
denfromufa opened this Issue Sep 6, 2015 · 46 comments

Comments

8 participants
@denfromufa
Member

denfromufa commented Sep 6, 2015

Looks like with CoreCLR the unmanaged exports "hack" for embedding CLR can be extended by cross-platform hosting API:

dotnet/coreclr#1256

This would allow simple C/C++ code to embed CLR

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Sep 6, 2015

Member

Note that for .NET 4+ still have to use unmanaged exports.

Member

denfromufa commented Sep 6, 2015

Note that for .NET 4+ still have to use unmanaged exports.

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Jul 22, 2016

Member

apparently @lstratman has extensive experience porting Mono to CoreCLR embedding.

Member

denfromufa commented Jul 22, 2016

apparently @lstratman has extensive experience porting Mono to CoreCLR embedding.

@lstratman

This comment has been minimized.

Show comment
Hide comment
@lstratman

lstratman Jul 25, 2016

Contributor

Thanks @denfromufa :).

It's funny, I actually had a fork of this repo from a while back with the goal of looking into this. But, since my company has only a single customer that might want to integrate our .NET server application into their Python application on Linux and, even then, only at some point in the future, it's down towards the bottom of my priority list for development unfortunately. However, I'm happy to offer some technical guidance.

I implemented basically this same functionality for Node.js for the Edge.js project and a lot of the code that I wrote there can probably be re-used or easily re-purposed for pythonnet. https://github.com/tjanczuk/edge/blob/master/src/CoreCLREmbedding/coreclrembedding.cpp#L192 contains the code that spins up the .NET Core CLR in any process. Towards the end of the method (https://github.com/tjanczuk/edge/blob/master/src/CoreCLREmbedding/coreclrembedding.cpp#L673), we get pointers to a bunch of .NET methods that serve as the managed end of the Node.js->.NET interop process. These function pointers are then invoked anytime that we need to call from Node.js to .NET. Implementations of those .NET methods are in https://github.com/tjanczuk/edge/blob/master/src/double/Edge.js/dotnetcore/coreclrembedding.cs and are included in the .NET Core application that's run from within Edge.js through a NuGet package that needs to be referenced (see https://github.com/tjanczuk/edge#how-to-specify-additional-clr-assembly-references-in-c-code).

This works well for Edge.js because the interaction between Node.js and .NET is all encapsulated in functions in one language or the other. So to call between the two domains, you're always invoking functions, never creating variables, setting properties, etc. However for pythonnet, we're obviously more "integrated" into the Python syntax, so we can do things like create variables, set properties, etc. You can write more methods in the .NET end of the interop process to facilitate this, or try to stay in unmanaged code as much as possible by calling the .NET Core CLR native methods to do things like create variables and set properties. That would be the more performant, but potentially more difficult, route. This will have to be done through the ICLRRuntimeHost instance that you get back from calling coreclr_initialize(). It should theoretically be possible, but is left as an exercise to the interested reader.

Anyway, that concludes my brain dump, I hope that it helps. Feel free to post questions and I'll do my best to answer them. At some point, I may have some time for actual development as well, but can't offer any promises.

Contributor

lstratman commented Jul 25, 2016

Thanks @denfromufa :).

It's funny, I actually had a fork of this repo from a while back with the goal of looking into this. But, since my company has only a single customer that might want to integrate our .NET server application into their Python application on Linux and, even then, only at some point in the future, it's down towards the bottom of my priority list for development unfortunately. However, I'm happy to offer some technical guidance.

I implemented basically this same functionality for Node.js for the Edge.js project and a lot of the code that I wrote there can probably be re-used or easily re-purposed for pythonnet. https://github.com/tjanczuk/edge/blob/master/src/CoreCLREmbedding/coreclrembedding.cpp#L192 contains the code that spins up the .NET Core CLR in any process. Towards the end of the method (https://github.com/tjanczuk/edge/blob/master/src/CoreCLREmbedding/coreclrembedding.cpp#L673), we get pointers to a bunch of .NET methods that serve as the managed end of the Node.js->.NET interop process. These function pointers are then invoked anytime that we need to call from Node.js to .NET. Implementations of those .NET methods are in https://github.com/tjanczuk/edge/blob/master/src/double/Edge.js/dotnetcore/coreclrembedding.cs and are included in the .NET Core application that's run from within Edge.js through a NuGet package that needs to be referenced (see https://github.com/tjanczuk/edge#how-to-specify-additional-clr-assembly-references-in-c-code).

This works well for Edge.js because the interaction between Node.js and .NET is all encapsulated in functions in one language or the other. So to call between the two domains, you're always invoking functions, never creating variables, setting properties, etc. However for pythonnet, we're obviously more "integrated" into the Python syntax, so we can do things like create variables, set properties, etc. You can write more methods in the .NET end of the interop process to facilitate this, or try to stay in unmanaged code as much as possible by calling the .NET Core CLR native methods to do things like create variables and set properties. That would be the more performant, but potentially more difficult, route. This will have to be done through the ICLRRuntimeHost instance that you get back from calling coreclr_initialize(). It should theoretically be possible, but is left as an exercise to the interested reader.

Anyway, that concludes my brain dump, I hope that it helps. Feel free to post questions and I'll do my best to answer them. At some point, I may have some time for actual development as well, but can't offer any promises.

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Sep 30, 2016

Member

@lstratman thank you for very detailed response, there is a question from MS if CoreCLR C-API is good enough, would you mind replying there? MS is considering standardizing C-API for .NET ecosystem.

https://blogs.msdn.microsoft.com/dotnet/2016/09/26/introducing-net-standard/#comment-138395

Are the four methods (Initialize, Shutdown, Create_Delegate, Execute_Assembly), mentioned enough for embedding CoreCLR in pythonnet?

Member

denfromufa commented Sep 30, 2016

@lstratman thank you for very detailed response, there is a question from MS if CoreCLR C-API is good enough, would you mind replying there? MS is considering standardizing C-API for .NET ecosystem.

https://blogs.msdn.microsoft.com/dotnet/2016/09/26/introducing-net-standard/#comment-138395

Are the four methods (Initialize, Shutdown, Create_Delegate, Execute_Assembly), mentioned enough for embedding CoreCLR in pythonnet?

@lstratman

This comment has been minimized.

Show comment
Hide comment
@lstratman

lstratman Oct 1, 2016

Contributor

I've replied to that thread that those methods are sufficient. Looking further into the existing pythonnet code, I realized that it does all of the things that I mentioned earlier (create variables, set properties, index arrays, etc.) in managed code anyway, so once the "spinning up CoreCLR" code is integrated from Edge.js, pythonnet should just be able to re-use a lot of its existing codebase.

Contributor

lstratman commented Oct 1, 2016

I've replied to that thread that those methods are sufficient. Looking further into the existing pythonnet code, I realized that it does all of the things that I mentioned earlier (create variables, set properties, index arrays, etc.) in managed code anyway, so once the "spinning up CoreCLR" code is integrated from Edge.js, pythonnet should just be able to re-use a lot of its existing codebase.

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Oct 5, 2016

Member

Another thing holding of CoreCLR port is Reflection.Emit used in pythonnet in one place:

dotnet/corefx#4491 (comment)

But we could possibly just do conditional compilation for this feature.

Member

denfromufa commented Oct 5, 2016

Another thing holding of CoreCLR port is Reflection.Emit used in pythonnet in one place:

dotnet/corefx#4491 (comment)

But we could possibly just do conditional compilation for this feature.

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Oct 5, 2016

Member

otherwise try ikvm.reflection.emit or mono.cecil

Member

denfromufa commented Oct 5, 2016

otherwise try ikvm.reflection.emit or mono.cecil

@var1ap

This comment has been minimized.

Show comment
Hide comment
@var1ap

var1ap Nov 23, 2016

Hi, I started work toward Python.Net Core edition in this fork https://github.com/var1ap/pythonnet , stay tuned.

var1ap commented Nov 23, 2016

Hi, I started work toward Python.Net Core edition in this fork https://github.com/var1ap/pythonnet , stay tuned.

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Dec 4, 2016

Member

@var1ap this is great news, do you agree to transition from Zope to MIT license, if your port is successful?

Member

denfromufa commented Dec 4, 2016

@var1ap this is great news, do you agree to transition from Zope to MIT license, if your port is successful?

@var1ap

This comment has been minimized.

Show comment
Hide comment
@var1ap

var1ap Dec 5, 2016

@denfromufa of course agree.

var1ap commented Dec 5, 2016

@denfromufa of course agree.

@var1ap

This comment has been minimized.

Show comment
Hide comment
@var1ap

var1ap Dec 5, 2016

Greate news!
var1ap@2b508fd
I managed to get pythonnet work under dotnet core 1.1(Win x64)

var1ap commented Dec 5, 2016

Greate news!
var1ap@2b508fd
I managed to get pythonnet work under dotnet core 1.1(Win x64)

@var1ap

This comment has been minimized.

Show comment
Hide comment
@var1ap

var1ap Dec 27, 2016

Also pythonnet successfully ported to dotnet core 1.1(Linux x64 Ubuntu 16.04) (in the same fork).
This is just Proof of concept. So somebody should port this project to CoreCLR with good code quality required for production projects.

var1ap commented Dec 27, 2016

Also pythonnet successfully ported to dotnet core 1.1(Linux x64 Ubuntu 16.04) (in the same fork).
This is just Proof of concept. So somebody should port this project to CoreCLR with good code quality required for production projects.

@dmitriyse

This comment has been minimized.

Show comment
Hide comment
@dmitriyse

dmitriyse Jan 1, 2017

Contributor

Hi guys! Pythonnet really works under dotnet core >= 1.0(NetStandard1.5). Thank you very much var1ap for your research. I continued to work on porting pythonnet to dotnet core in this repo https://github.com/dmitriyse/pythonnet/tree/coreclr. Also I plan to improve my binary distribution and avoid any dll load magic. Finally we should get good nuget based distribution.

Contributor

dmitriyse commented Jan 1, 2017

Hi guys! Pythonnet really works under dotnet core >= 1.0(NetStandard1.5). Thank you very much var1ap for your research. I continued to work on porting pythonnet to dotnet core in this repo https://github.com/dmitriyse/pythonnet/tree/coreclr. Also I plan to improve my binary distribution and avoid any dll load magic. Finally we should get good nuget based distribution.

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Feb 22, 2017

Member

@var1ap @dmitriyse can you prepare a pull request with compilation flags for coreclr? much like .NET/Mono flags that we have currently.

Member

denfromufa commented Feb 22, 2017

@var1ap @dmitriyse can you prepare a pull request with compilation flags for coreclr? much like .NET/Mono flags that we have currently.

@dmitriyse

This comment has been minimized.

Show comment
Hide comment
@dmitriyse

dmitriyse Feb 22, 2017

Contributor

I can spend time to reapply coreclr code adoption in the next week.
Please review talk in those posts dmitriyse@38adb2e#commitcomment-20774083

Please consider with vmuriart, which fork/branch is most suitable for this work, where I should fork.

Contributor

dmitriyse commented Feb 22, 2017

I can spend time to reapply coreclr code adoption in the next week.
Please review talk in those posts dmitriyse@38adb2e#commitcomment-20774083

Please consider with vmuriart, which fork/branch is most suitable for this work, where I should fork.

@vmuriart

This comment has been minimized.

Show comment
Hide comment
@vmuriart

vmuriart Feb 22, 2017

Contributor

@dmitriyse on top of master, but at the rate we are committing changes at the moment it wouldn't make sense to do all the work if it won't get merged in immediately.

@denfromufa, after reviewing @dmitriyse and @var1ap, their changes look pretty straight forward to upgrade to coreclr. My hesitation is that it seems that it would require us to upgrade from 4.0 to 4.5 framework which may drop users (I still use 4.0). We could throw in a bunch of #if directives, but it would make the code very messy. (side note, We actually got rid of most mono_linux and mono_osx flags from the code thanks to @dmitriyse)

Another issue is that coreclr is still undergoing changes wrt their build system. They've gone from project.json back to an upgraded msbuild with new project file formats. mono wouldn't compile against either of these options; ie. we would need to have two sets of csproj/sln/packages.config files in order to ensure mono can still be used for compiling (unless we drop mono?)

Currently what I'm doing is slowly porting over some of the changes @dmitriyse and @var1ap did to minimize the changes for when we migrate to coreclr. I'm hoping to find a clean way to maintain clr 4.0 and add coreclr, but at the moment I'm not finding it and it might need to be part of pythonnet 3.0.

Contributor

vmuriart commented Feb 22, 2017

@dmitriyse on top of master, but at the rate we are committing changes at the moment it wouldn't make sense to do all the work if it won't get merged in immediately.

@denfromufa, after reviewing @dmitriyse and @var1ap, their changes look pretty straight forward to upgrade to coreclr. My hesitation is that it seems that it would require us to upgrade from 4.0 to 4.5 framework which may drop users (I still use 4.0). We could throw in a bunch of #if directives, but it would make the code very messy. (side note, We actually got rid of most mono_linux and mono_osx flags from the code thanks to @dmitriyse)

Another issue is that coreclr is still undergoing changes wrt their build system. They've gone from project.json back to an upgraded msbuild with new project file formats. mono wouldn't compile against either of these options; ie. we would need to have two sets of csproj/sln/packages.config files in order to ensure mono can still be used for compiling (unless we drop mono?)

Currently what I'm doing is slowly porting over some of the changes @dmitriyse and @var1ap did to minimize the changes for when we migrate to coreclr. I'm hoping to find a clean way to maintain clr 4.0 and add coreclr, but at the moment I'm not finding it and it might need to be part of pythonnet 3.0.

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Feb 22, 2017

Member
Member

denfromufa commented Feb 22, 2017

@dmitriyse

This comment has been minimized.

Show comment
Hide comment
@dmitriyse

dmitriyse Mar 28, 2017

Contributor

Hi! I restarted work on porting PythonNet to CoreCLR. Now it can be performed as additional files.
So current build and new build (based on .csproj 2017) can coexist side-by-side.
https://github.com/dmitriyse/pythonnet/tree/coreclr

Contributor

dmitriyse commented Mar 28, 2017

Hi! I restarted work on porting PythonNet to CoreCLR. Now it can be performed as additional files.
So current build and new build (based on .csproj 2017) can coexist side-by-side.
https://github.com/dmitriyse/pythonnet/tree/coreclr

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Mar 28, 2017

Member

@dmitriyse this looks much cleaner! Can you open a pull request for review cycle?

image

Member

denfromufa commented Mar 28, 2017

@dmitriyse this looks much cleaner! Can you open a pull request for review cycle?

image

@dmitriyse

This comment has been minimized.

Show comment
Hide comment
@dmitriyse

dmitriyse Mar 28, 2017

Contributor

Ok, I will create pool request right after some useful point. I fixed 102 from 120 migration issues to get pythonnet working with at least one build configuration.

Contributor

dmitriyse commented Mar 28, 2017

Ok, I will create pool request right after some useful point. I fixed 102 from 120 migration issues to get pythonnet working with at least one build configuration.

@dmitriyse

This comment has been minimized.

Show comment
Hide comment
@dmitriyse

dmitriyse Sep 6, 2017

Contributor

Half of the problem solved - python can be embedded in to CoreCLR 2.0 (#519)

System.Reflection.Emit works good under CoreCLR 2.0 except for CDecl calling convention. CDecl currently unsupported in the CoreCLR 2.0

As a workaround I changed XDecref/XIncref to Py_DecRef/, Py_IncRef.
And used UnamanagedFunctionPoinerAttribute with Marshal.GetDelegateForFunctionPointer, that works good, and does not looks too slow.

The rest part is to inject CoreCLR into python through CoreCLR C Api.

Contributor

dmitriyse commented Sep 6, 2017

Half of the problem solved - python can be embedded in to CoreCLR 2.0 (#519)

System.Reflection.Emit works good under CoreCLR 2.0 except for CDecl calling convention. CDecl currently unsupported in the CoreCLR 2.0

As a workaround I changed XDecref/XIncref to Py_DecRef/, Py_IncRef.
And used UnamanagedFunctionPoinerAttribute with Marshal.GetDelegateForFunctionPointer, that works good, and does not looks too slow.

The rest part is to inject CoreCLR into python through CoreCLR C Api.

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Sep 7, 2017

Member

@dmitriyse for cdecl did you mean this issue? dotnet/corefx#9800

Member

denfromufa commented Sep 7, 2017

@dmitriyse for cdecl did you mean this issue? dotnet/corefx#9800

@dmitriyse

This comment has been minimized.

Show comment
Hide comment
@dmitriyse

dmitriyse Sep 7, 2017

Contributor

Yes, exactly this flaw.

Contributor

dmitriyse commented Sep 7, 2017

Yes, exactly this flaw.

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Sep 13, 2017

Member

@dmitriyse for rest part to inject CoreCLR into Python we can avoid CoreCLR C API, if the DLLExport becomes stable with .NET Core, @3F has to use custom CoreCLR at the moment. DLLExport is the next generation of UnmanagedExports, that we are currently using in .NET Framework:

https://github.com/3F/DllExport

Member

denfromufa commented Sep 13, 2017

@dmitriyse for rest part to inject CoreCLR into Python we can avoid CoreCLR C API, if the DLLExport becomes stable with .NET Core, @3F has to use custom CoreCLR at the moment. DLLExport is the next generation of UnmanagedExports, that we are currently using in .NET Framework:

https://github.com/3F/DllExport

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Sep 14, 2017

Member

@dmitriyse with your valid finalizer from #532 I'm getting this unexpected crash on .NET Core 2.0, which does not appear on .NET Framework:

C:\Users\denis.akhiyarov\Downloads\pythonnet-valid-finalizer\build\lib.win32-2.7\netcoreapp2.0>dotnet nPython.dll
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import clr
>>> import System
>>> System.Double(1.0)

Unhandled Exception: System.InvalidCastException: Unable to cast object of type 'TernaryFunc' to type 'IntPtr_3_Delegate'.
   at Python.Runtime.NativeCall.Call_3(IntPtr fp, IntPtr a1, IntPtr a2, IntPtr a3)
   at Python.Runtime.MetaType.tp_call(IntPtr tp, IntPtr args, IntPtr kw)
   at Python.Runtime.Runtime.Py_Main(Int32 argc, String[] argv)
   at Python.Runtime.PythonConsole.Main(String[] args)

C:\Users\denis.akhiyarov\Downloads\pythonnet-valid-finalizer\build\lib.win32-2.7\netcoreapp2.0>cd..

C:\Users\denis.akhiyarov\Downloads\pythonnet-valid-finalizer\build\lib.win32-2.7>nPython
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import clr
>>> import System
>>> System.Double(1.0)
<System.Double object at 0x052510B0>
Member

denfromufa commented Sep 14, 2017

@dmitriyse with your valid finalizer from #532 I'm getting this unexpected crash on .NET Core 2.0, which does not appear on .NET Framework:

C:\Users\denis.akhiyarov\Downloads\pythonnet-valid-finalizer\build\lib.win32-2.7\netcoreapp2.0>dotnet nPython.dll
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import clr
>>> import System
>>> System.Double(1.0)

Unhandled Exception: System.InvalidCastException: Unable to cast object of type 'TernaryFunc' to type 'IntPtr_3_Delegate'.
   at Python.Runtime.NativeCall.Call_3(IntPtr fp, IntPtr a1, IntPtr a2, IntPtr a3)
   at Python.Runtime.MetaType.tp_call(IntPtr tp, IntPtr args, IntPtr kw)
   at Python.Runtime.Runtime.Py_Main(Int32 argc, String[] argv)
   at Python.Runtime.PythonConsole.Main(String[] args)

C:\Users\denis.akhiyarov\Downloads\pythonnet-valid-finalizer\build\lib.win32-2.7\netcoreapp2.0>cd..

C:\Users\denis.akhiyarov\Downloads\pythonnet-valid-finalizer\build\lib.win32-2.7>nPython
Python 2.7.13 (v2.7.13:a06454b1afa1, Dec 17 2016, 20:42:59) [MSC v.1500 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import clr
>>> import System
>>> System.Double(1.0)
<System.Double object at 0x052510B0>
@dmitriyse

This comment has been minimized.

Show comment
Hide comment
@dmitriyse

dmitriyse Sep 15, 2017

Contributor

Thank you for the case. I will try to reproduce and debug.

Contributor

dmitriyse commented Sep 15, 2017

Thank you for the case. I will try to reproduce and debug.

@dmitriyse

This comment has been minimized.

Show comment
Hide comment
@dmitriyse

dmitriyse Oct 2, 2017

Contributor

I found the problem and solution. It's a DotNet core bug. I will create PR soon.

Contributor

dmitriyse commented Oct 2, 2017

I found the problem and solution. It's a DotNet core bug. I will create PR soon.

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Oct 8, 2017

Member

some progress on .NET Core version of DllExport:

3F/DllExport#17 (comment)

Member

denfromufa commented Oct 8, 2017

some progress on .NET Core version of DllExport:

3F/DllExport#17 (comment)

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Oct 8, 2017

Member

@dmitriyse another approach we can take (if DLLExport has technical limitations, and C-API is too hard), is using Embeddinator project from Mono, which supposedly also works under .NET Standard / .NET Core:

mono/Embeddinator-4000#515

Member

denfromufa commented Oct 8, 2017

@dmitriyse another approach we can take (if DLLExport has technical limitations, and C-API is too hard), is using Embeddinator project from Mono, which supposedly also works under .NET Standard / .NET Core:

mono/Embeddinator-4000#515

@dmitriyse

This comment has been minimized.

Show comment
Hide comment
@dmitriyse

dmitriyse Oct 8, 2017

Contributor

I am afraid, that Embeddinator-4000 uses mono/.net framework under the hood. We need to inspect example with .Net Standard 1.4 library to ensure that DotNotCore 2.0 host can be used.

Contributor

dmitriyse commented Oct 8, 2017

I am afraid, that Embeddinator-4000 uses mono/.net framework under the hood. We need to inspect example with .Net Standard 1.4 library to ensure that DotNotCore 2.0 host can be used.

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Oct 8, 2017

Member

@dmitriyse here is why I think this is possible:

mono/CppSharp#613

Member

denfromufa commented Oct 8, 2017

@dmitriyse here is why I think this is possible:

mono/CppSharp#613

@Cronan

This comment has been minimized.

Show comment
Hide comment
@Cronan

Cronan Jan 8, 2018

Contributor

Any updates on this issue? As per a discussion with @denfromufa, it seems this might hold the key: https://developer.xamarin.com/guides/cross-platform/dotnet-embedding/getting-started/c/

Contributor

Cronan commented Jan 8, 2018

Any updates on this issue? As per a discussion with @denfromufa, it seems this might hold the key: https://developer.xamarin.com/guides/cross-platform/dotnet-embedding/getting-started/c/

@denfromufa denfromufa changed the title from CoreCLR embedding - cross-platform API to .NET Core support and CoreCLR embedding - cross-platform API Jan 18, 2018

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Jan 29, 2018

Member

@Cronan very disappointing, but I just found out that .NET Core is not implemented in embeddinator. Hence we need to use this cross-platform C/C++ hosting API for .NET Core like it is currently done for Mono:

https://docs.microsoft.com/en-us/dotnet/core/tutorials/netcore-hosting

Member

denfromufa commented Jan 29, 2018

@Cronan very disappointing, but I just found out that .NET Core is not implemented in embeddinator. Hence we need to use this cross-platform C/C++ hosting API for .NET Core like it is currently done for Mono:

https://docs.microsoft.com/en-us/dotnet/core/tutorials/netcore-hosting

@Cronan

This comment has been minimized.

Show comment
Hide comment
@Cronan

Cronan Jan 30, 2018

Contributor

@denfromufa @dmitriyse I've got a stand-alone test project that successfully calls a method in a dotnet core 2.0 app, using the cross-platform C/C++ hosting API for .Net Core 2.0. The code uses dlopen() and dlsym() to initialize the core CLR runtime, and calls a function using a delegate.

It's not really a branch, more of a stand-alone piece of work for now, but I hope it won't be too much work to integrate into the library.

Where should I put it so you can take a look?

Contributor

Cronan commented Jan 30, 2018

@denfromufa @dmitriyse I've got a stand-alone test project that successfully calls a method in a dotnet core 2.0 app, using the cross-platform C/C++ hosting API for .Net Core 2.0. The code uses dlopen() and dlsym() to initialize the core CLR runtime, and calls a function using a delegate.

It's not really a branch, more of a stand-alone piece of work for now, but I hope it won't be too much work to integrate into the library.

Where should I put it so you can take a look?

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Jan 30, 2018

Member
Member

denfromufa commented Jan 30, 2018

@Cronan

This comment has been minimized.

Show comment
Hide comment
@Cronan

Cronan Jan 31, 2018

Contributor

Build

dotnet build ./dotnet
python setup.py build_ext --inplace

Run

python
>>> import clr
>>> clr.sum(4,5)
9.0

coreclr_host.zip

Contributor

Cronan commented Jan 31, 2018

Build

dotnet build ./dotnet
python setup.py build_ext --inplace

Run

python
>>> import clr
>>> clr.sum(4,5)
9.0

coreclr_host.zip

@djoyce82

This comment has been minimized.

Show comment
Hide comment
@djoyce82

djoyce82 Jan 31, 2018

@denfromufa @Cronan I have been working on a fork of this here:

https://github.com/djoyce82/pythonnet

It's very similar to what @Cronan has put together (similar reference material :-) ) but I have ported it to C as this is what the current mono implementation appears to be targeting.

I've tested this on Linux (CentOS 7) and the code works for my use case.

Would you like me to create a pull request for these changes?

djoyce82 commented Jan 31, 2018

@denfromufa @Cronan I have been working on a fork of this here:

https://github.com/djoyce82/pythonnet

It's very similar to what @Cronan has put together (similar reference material :-) ) but I have ported it to C as this is what the current mono implementation appears to be targeting.

I've tested this on Linux (CentOS 7) and the code works for my use case.

Would you like me to create a pull request for these changes?

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Jan 31, 2018

Member

@djoyce82 this is great! definitely, please submit a pull request.

Member

denfromufa commented Jan 31, 2018

@djoyce82 this is great! definitely, please submit a pull request.

@Cronan

This comment has been minimized.

Show comment
Hide comment
@Cronan

Cronan Jan 31, 2018

Contributor

@djoyce82 That's great work, does it build and run from Linux? What commands did you use to build it and run the tests?

Contributor

Cronan commented Jan 31, 2018

@djoyce82 That's great work, does it build and run from Linux? What commands did you use to build it and run the tests?

@Cronan

This comment has been minimized.

Show comment
Hide comment
@Cronan

Cronan Jan 31, 2018

Contributor

@djoyce82

python setup.py bdist_wheel --xplat
Microsoft (R) Build Engine version 15.3.409.57025 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Python.Runtime.15 -> /users/isys/icronyn/git/pythonnet_clr/src/runtime/bin/netstandard2.0/Python.Runtime.dll
  Console.15 -> /users/isys/icronyn/git/pythonnet_clr/src/console/bin/netcoreapp2.0/nPython.dll
  Console.15 -> /users/isys/icronyn/git/pythonnet_clr/build/lib.linux-x86_64-2.7/netcoreapp2.0/
  Python.EmbeddingTest.15 -> /users/isys/icronyn/git/pythonnet_clr/src/embed_tests/bin/netcoreapp2.0/Python.EmbeddingTest.dll
  Python.EmbeddingTest.15 -> /users/isys/icronyn/git/pythonnet_clr/src/embed_tests/bin/netcoreapp2.0_publish/
building 'clr' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
creating build/temp.linux-x86_64-2.7/src/coreclr
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/man/releases/python-medusa/27-1/include/python2.7 -c src/coreclr/pynetinit.c -o build/temp.linux-x86_64-2.7/src/coreclr/pynetinit.o
src/coreclr/pynetinit.c: In function \u2018init\u2019:
src/coreclr/pynetinit.c:112:5: error: \u2018for\u2019 loop initial declarations are only allowed in C99 mode
     for (int ii = 0; ii < PyList_Size(syspath); ++ii)

Contributor

Cronan commented Jan 31, 2018

@djoyce82

python setup.py bdist_wheel --xplat
Microsoft (R) Build Engine version 15.3.409.57025 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

  Python.Runtime.15 -> /users/isys/icronyn/git/pythonnet_clr/src/runtime/bin/netstandard2.0/Python.Runtime.dll
  Console.15 -> /users/isys/icronyn/git/pythonnet_clr/src/console/bin/netcoreapp2.0/nPython.dll
  Console.15 -> /users/isys/icronyn/git/pythonnet_clr/build/lib.linux-x86_64-2.7/netcoreapp2.0/
  Python.EmbeddingTest.15 -> /users/isys/icronyn/git/pythonnet_clr/src/embed_tests/bin/netcoreapp2.0/Python.EmbeddingTest.dll
  Python.EmbeddingTest.15 -> /users/isys/icronyn/git/pythonnet_clr/src/embed_tests/bin/netcoreapp2.0_publish/
building 'clr' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/src
creating build/temp.linux-x86_64-2.7/src/coreclr
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/opt/man/releases/python-medusa/27-1/include/python2.7 -c src/coreclr/pynetinit.c -o build/temp.linux-x86_64-2.7/src/coreclr/pynetinit.o
src/coreclr/pynetinit.c: In function \u2018init\u2019:
src/coreclr/pynetinit.c:112:5: error: \u2018for\u2019 loop initial declarations are only allowed in C99 mode
     for (int ii = 0; ii < PyList_Size(syspath); ++ii)

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa
Member

denfromufa commented Jan 31, 2018

@Cronan pass '-std=c99' to compile args:

https://stackoverflow.com/a/10867041/2230844

@Cronan

This comment has been minimized.

Show comment
Hide comment
@Cronan

Cronan Jan 31, 2018

Contributor

Thanks @denfromufa, I fixed it by declaring the variable outside the loop - there was a single other case of this:

src/coreclr/coreutils.c: In function \u2018AddFilesFromDirectoryToTpaList\u2019:
src/coreclr/coreutils.c:236:5: error: \u2018for\u2019 loop initial declarations are only allowed in C99 mode
     for (size_t extIndex = 0; extIndex < sizeof(tpaExtensions) / sizeof(tpaExtensions[0]); extIndex++)
     ^
src/coreclr/coreutils.c:236:5: note: use option -std=c99 or -std=gnu99 to compile your code
error: command 'gcc' failed with exit status 1
Contributor

Cronan commented Jan 31, 2018

Thanks @denfromufa, I fixed it by declaring the variable outside the loop - there was a single other case of this:

src/coreclr/coreutils.c: In function \u2018AddFilesFromDirectoryToTpaList\u2019:
src/coreclr/coreutils.c:236:5: error: \u2018for\u2019 loop initial declarations are only allowed in C99 mode
     for (size_t extIndex = 0; extIndex < sizeof(tpaExtensions) / sizeof(tpaExtensions[0]); extIndex++)
     ^
src/coreclr/coreutils.c:236:5: note: use option -std=c99 or -std=gnu99 to compile your code
error: command 'gcc' failed with exit status 1
@Cronan

This comment has been minimized.

Show comment
Hide comment
@Cronan

Cronan Jan 31, 2018

Contributor

@denfromufa @djoyce82 Lots of the tests failed, with this kind of thing:

Unhandled Exception: System.MissingMethodException: Constructor on type 'System.Reflection.Emit.TypeBuilder' not found.
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Activator.CreateInstance(Type type, Object[] args)
   at Python.Runtime.DelegateManager.GetDelegate(Type dtype, IntPtr callable)
   at Python.Runtime.EventObject.AddEventHandler(IntPtr target, IntPtr handler)
   at Python.Runtime.EventBinding.nb_inplace_add(IntPtr ob, IntPtr arg)
Aborted
Contributor

Cronan commented Jan 31, 2018

@denfromufa @djoyce82 Lots of the tests failed, with this kind of thing:

Unhandled Exception: System.MissingMethodException: Constructor on type 'System.Reflection.Emit.TypeBuilder' not found.
   at System.RuntimeType.CreateInstanceImpl(BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes, StackCrawlMark& stackMark)
   at System.Activator.CreateInstance(Type type, BindingFlags bindingAttr, Binder binder, Object[] args, CultureInfo culture, Object[] activationAttributes)
   at System.Activator.CreateInstance(Type type, Object[] args)
   at Python.Runtime.DelegateManager.GetDelegate(Type dtype, IntPtr callable)
   at Python.Runtime.EventObject.AddEventHandler(IntPtr target, IntPtr handler)
   at Python.Runtime.EventBinding.nb_inplace_add(IntPtr ob, IntPtr arg)
Aborted

@djoyce82 djoyce82 referenced this issue Jan 31, 2018

Open

Initial coreclr and build tooling #612

0 of 4 tasks complete
@djoyce82

This comment has been minimized.

Show comment
Hide comment
@djoyce82

djoyce82 Jan 31, 2018

@Cronan I believe the test failures are the same issue as detailed in #590 and #571, looks like the implementation of System.Reflection.Emit.TypeBuilder has changed in .NET Core?

PR is #612

djoyce82 commented Jan 31, 2018

@Cronan I believe the test failures are the same issue as detailed in #590 and #571, looks like the implementation of System.Reflection.Emit.TypeBuilder has changed in .NET Core?

PR is #612

@weixu02

This comment has been minimized.

Show comment
Hide comment
@weixu02

weixu02 Aug 3, 2018

If I want to use a .net core assembly in python and only care about using on windows, is that possible with the current python.net version? Thanks!

weixu02 commented Aug 3, 2018

If I want to use a .net core assembly in python and only care about using on windows, is that possible with the current python.net version? Thanks!

@denfromufa

This comment has been minimized.

Show comment
Hide comment
@denfromufa

denfromufa Aug 23, 2018

Member

@weixu02 this is possible via npython.exe if you compile the full pythonnet solution

Member

denfromufa commented Aug 23, 2018

@weixu02 this is possible via npython.exe if you compile the full pythonnet solution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment