macOS env #2

lokinfey · 2019-06-22T09:18:11Z

I try to use Numpy.NET in my macos

but it give me this error

Unable to load shared library 'python36' or one of its dependencies. In order to help diagnose loading problems, consider setting the DYLD_PRINT_LIBRARIES environment variable: dlopen(libpython36, 1): image not found at Python.Runtime.Runtime.Py_IsInitialized() at Python.Runtime.Runtime.Initialize(Boolean initSigs) at Python.Runtime.PythonEngine.Initialize(IEnumerable1 args, Boolean setSysArgv, Boolean initSigs) at Numpy.NumPy.InstallAndImport(Boolean force) at Numpy.NumPy.<>c.<.cctor>b__650_0() at System.Lazy1.ViaFactory(LazyThreadSafetyMode mode) at System.Lazy1.ExecutionAndPublication(LazyHelper executionAndPublication, Boolean useDefaultConstructor) at System.Lazy1.CreateValue() at Numpy.np.array[T](T[,] object, Dtype dtype, Nullable1 copy, String order, Nullable1 subok, Nullable1 ndmin) at Submission#4.<>d__0.MoveNext() in :line 1 --- End of stack trace from previous location where exception was thrown --- at Microsoft.CodeAnalysis.Scripting.ScriptExecutionState.RunSubmissionsAsync[TResult](ImmutableArray1 precedingExecutors, Func2 currentExecutor, StrongBox1 exceptionHolderOpt, Func2 catchExceptionOpt, CancellationToken cancellationToken) at Microsoft.CodeAnalysis.Scripting.Script1.RunSubmissionsAsync(ScriptExecutionState executionState, ImmutableArray1 precedingExecutors, Func2 currentExecutor, Func`2 catchExceptionOpt, CancellationToken cancellationToken) at ICSharpCore.Script.InteractiveScriptEngine.ExecuteAsync(String statement)

The text was updated successfully, but these errors were encountered:

henon · 2019-06-22T10:36:00Z

I don't own a mac so it is not easy for me to test with your scenario. Are you using Numpy.Bare? My guess is that pythonnet is compiled against a windows binary? Could you try to run the simple example from the pythonnet readme on macOS to check if you get the same error message?

If you are willing to work on macOS support I will support you with my knowledge.

henon · 2019-06-22T10:43:03Z

One more thing:
If you use Numpy.Bare.dll it doesn't reference Python.Included, just pythonnet_netstandard.
If you use Numpy.dll it uses Python.Included which in turn references pythonnet_netstandard.

Step one is to find out if the problem is at pythonnet_netstandard or at Python.Included.

lokinfey · 2019-06-23T01:22:37Z

I change Numpy.Bare.dll is the same problem

henon · 2019-06-23T09:52:47Z

@lokinfey Are you willing to help with this issue?

Nucs · 2019-06-23T10:53:44Z

The issue is that operating system is trying to find the installed python 3.6 on the machine.
As the exception tells you, "consider setting the DYLD_PRINT_LIBRARIES environment variable" to the Python\Python36\Lib directory.

More about the DYLD_PRINT_LIBRARIES.
A similar issue in pythonnet #767 - he has a detailed solution.
Let me know if it helped you as I can't test it myself.

deepakkumar1984 · 2019-06-23T12:19:37Z

I have MAC, i will try out Nucs suggestion. But I think we need to build Numpy with Pythonnet with OSX flag which will pick the dylib files. Other test I can is using dll.config file as per this https://answers.unity.com/questions/359991/how-to-use-mono-dllnamedllconfig-with-unity.html

henon · 2019-06-23T12:45:45Z

@deepakkumar1984 thanks man. Make sure you use my fork of pythonnet because it has some additions that were necessary for Numpy. By the way, for Keras.NET it is also the correct dependency, not Python.Included, like you wrote.

Oceania2018 · 2019-06-23T13:46:29Z

Related issue pythonnet/pythonnet#892

@deepakkumar1984 Does this help? http://curtis.schlak.com/2012/01/17/howto-run-pythonnet-on-osx.html

deepakkumar1984 · 2019-06-23T23:40:17Z

The Numpy project is built with pythonnet with WIN parameters, which will look for Python37.dll. After building with OSX parameters, it's failing at method "PyUnicode_FromKindAndData" in runtime.cs:

public static IntPtr PyString_FromString(string value)
        {
#if PYTHON3
            return PyUnicode_FromKindAndData(_UCS, value, value.Length);
#elif PYTHON2
            return PyString_FromStringAndSize(value, value.Length);
#endif
        }

Had to make the following change in the runtime.cs:

#if MONO_LINUX // Linux/macOS use dotted version string
        internal const string dllBase = "python" + _pyversion;
#elif MONO_OSX // Linux/macOS use dotted version string
        internal const string dllBase = "libpython"; //The change to load python in MAC
#else // Windows
        internal const string dllBase = "python" + _pyver;
#endif

In Initialize() method:

//In OSX python PyObject_GetAttrString(op, "__dict__") chashes. Had to add condition to use Python2 method
if (IsPython3 && Environment.OSVersion.Platform.ToString() != "Unix")
            {
                op = PyImport_ImportModule("builtins");
                dict = PyObject_GetAttrString(op, "__dict__");
            }
            else// Python2
            {
                dict = PyImport_GetModuleDict();
                op = PyDict_GetItemString(dict, "__builtin__");
            }

I am still looking into the error with Unicode method if anyone knows please comment.

Nucs · 2019-06-24T00:08:34Z

That's weird, are you absolutely sure you are using python 3+?

PY3 - PyUnicode_FromKindAndData
PY2 - PyString_FromStringAndSize

Edit: If it is possible in osx to monitor process's file accessing, you should verify it using it.

deepakkumar1984 · 2019-06-24T00:29:36Z

Yes I have Python 3.7 installed. It doesn't work either "PyUnicode_FromKindAndData" and "PyString_FromStringAndSize". So I am not sure if there is 3rd function specific for OSX

Nucs · 2019-06-24T00:32:10Z

Can you post here the exact error? If it fails both methods, it seems to have failed loading the library dlls

deepakkumar1984 · 2019-06-24T00:37:33Z

I will post the full stack after getting back home (no mac at workplace), It seems to load the Python dlls and invoked few Initialization methods in the Runtime.Initialize method and failed at line 337: op = PyString_FromString("string");

deepakkumar1984 · 2019-06-25T11:08:26Z

Stack Trace:
System.EntryPointNotFoundException: "Unable to find an entry point named 'PyUnicode_FromKindAndData' in shared library 'libpython'."
at Python.Runtime.Runtime.PyUnicode_FromKindAndData(Int32 kind, String s, IntPtr size)\n at Python.Runtime.Runtime.PyUnicode_FromKindAndData(Int32 kind, String s, Int64 size) in /Users/deepakbattini/work/git/pythonnet/src/runtime/runtime.cs:line 1464\n at Python.Runtime.Runtime.PyString_FromString(String value) in /Users/deepakbattini/work/git/pythonnet/src/runtime/runtime.cs:line 1392\n at Python.Runtime.Runtime.Initialize(Boolean initSigs) in /Users/deepakbattini/work/git/pythonnet/src/runtime/runtime.cs:line 341\n at Python.Runtime.PythonEngine.Initialize(IEnumerable`1 args, Boolean setSysArgv, Boolean initSigs) in /Users/deepakbattini/work/git/pythonnet/src/runtime/pythonengine.cs:line 168\n at Python.Runtime.PythonEngine.Initialize(Boolean setSysArgv, Boolean initSigs) in /Users/deepakbattini/work/git/pythonnet/src/runtime/pythonengine.cs:line 145\n at Python.Runtime.PythonEngine.Initialize() in /Users/deepakbattini/work/git/pythonnet/src/runtime/pythonengine.cs:line 140\n at Python.Runtime.PythonConsole.Main(String[] args) in /Users/deepakbattini/work/git/pythonnet/src/console/pythonconsole.cs:35

Nucs · 2019-06-25T11:22:26Z

Give this a shot, the other answers are are too relevant.

deepakkumar1984 · 2019-06-25T12:39:51Z

I configured the Console app with DYLD_PRINT_LIBRARIES to print all the loaded binaries. Although Python3.7 is installed, but the console shows python2.7
dyld: loaded: /System/Library/Frameworks/Python.framework/Versions/2.7/Python
dyld: loaded: /System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload/_locale.so

By default python3 shows up by running the command. And I cannot uninstall Python2.7 because it comes with OSX and will corrupt the OS. I am looking for options but looks like this is the main issue.

henon · 2019-06-25T14:07:24Z

if it is something fundamental like you suggest others must have similar problems with python3

lokinfey · 2019-06-30T12:38:21Z

The Numpy project is built with pythonnet with WIN parameters, which will look for Python37.dll. After building with OSX parameters, it's failing at method "PyUnicode_FromKindAndData" in runtime.cs:

public static IntPtr PyString_FromString(string value)
        {
#if PYTHON3
            return PyUnicode_FromKindAndData(_UCS, value, value.Length);
#elif PYTHON2
            return PyString_FromStringAndSize(value, value.Length);
#endif
        }

Had to make the following change in the runtime.cs:

#if MONO_LINUX // Linux/macOS use dotted version string
        internal const string dllBase = "python" + _pyversion;
#elif MONO_OSX // Linux/macOS use dotted version string
        internal const string dllBase = "libpython"; //The change to load python in MAC
#else // Windows
        internal const string dllBase = "python" + _pyver;
#endif

In Initialize() method:

//In OSX python PyObject_GetAttrString(op, "__dict__") chashes. Had to add condition to use Python2 method
if (IsPython3 && Environment.OSVersion.Platform.ToString() != "Unix")
            {
                op = PyImport_ImportModule("builtins");
                dict = PyObject_GetAttrString(op, "__dict__");
            }
            else// Python2
            {
                dict = PyImport_GetModuleDict();
                op = PyDict_GetItemString(dict, "__builtin__");
            }

I am still looking into the error with Unicode method if anyone knows please comment.

But when I followed all the step in wiki ,and set the Python.Runtime.dll.config , I can run but no results and no some errors . I think libpython3.7m.dylib is the problem about that ........ Can you give me some suggestions?

henon · 2019-06-30T12:44:24Z

I suggest creating an issue on the pythonnet repo. There the probability that someone can help with the problem is highest. But make sure you post a pure example without any mention of Numpy.NET in order to avoid confusion.

Oceania2018 · 2019-07-01T12:10:09Z

@henon FYI pythonnet/pythonnet#903

Esther2013 · 2019-07-03T02:09:31Z

@henon I tried it on my macOS, it's passed when I set the compile condition as MONO_OSX.

Here is the define symbols: TRACE;RELEASE;NETSTANDARD;NETSTANDARD2_0;MONO_OSX.

Test run for /Users/estherhu/Projects/pythonnet/src/embed_tests/bin/Debug/netcoreapp2.2/Python.EmbeddingTest.dll(.NETCoreApp,Version=v2.2)
Microsoft (R) Test Execution Command Line Tool Version 15.9.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Failed   TestCastGlobalVar
Error Message:
 Python.Runtime.PythonException : ImportError : No module named PyImportTest.cast_global_var
Stack Trace:
   at Python.Runtime.Runtime.CheckExceptionOccurred() in /Users/estherhu/Projects/pythonnet/src/runtime/runtime.cs:line 569
   at Python.Runtime.PythonEngine.ImportModule(String name) in /Users/estherhu/Projects/pythonnet/src/runtime/pythonengine.cs:line 469
   at Python.Runtime.Py.Import(String name) in /Users/estherhu/Projects/pythonnet/src/runtime/pythonengine.cs:line 693
   at Python.EmbeddingTest.PyImportTest.TestCastGlobalVar() in /Users/estherhu/Projects/pythonnet/src/embed_tests/pyimport.cs:line 78
Failed   TestDottedName
Error Message:
 Python.Runtime.PythonException : ImportError : No module named PyImportTest.test.one
Stack Trace:
   at Python.Runtime.Runtime.CheckExceptionOccurred() in /Users/estherhu/Projects/pythonnet/src/runtime/runtime.cs:line 569
   at Python.Runtime.PythonEngine.ImportModule(String name) in /Users/estherhu/Projects/pythonnet/src/runtime/pythonengine.cs:line 469
   at Python.EmbeddingTest.PyImportTest.TestDottedName() in /Users/estherhu/Projects/pythonnet/src/embed_tests/pyimport.cs:line 58
Failed   TestSysArgsImportException
Error Message:
 Python.Runtime.PythonException : ImportError : No module named PyImportTest.sysargv
Stack Trace:
   at Python.Runtime.Runtime.CheckExceptionOccurred() in /Users/estherhu/Projects/pythonnet/src/runtime/runtime.cs:line 569
   at Python.Runtime.PythonEngine.ImportModule(String name) in /Users/estherhu/Projects/pythonnet/src/runtime/pythonengine.cs:line 469
   at Python.EmbeddingTest.PyImportTest.TestSysArgsImportException() in /Users/estherhu/Projects/pythonnet/src/embed_tests/pyimport.cs:line 68
Skipped  ReInitialize
Skipped  SimpleTestMemory
Skipped  TestUnicode
Skipped  IsStringTrue
Skipped  TestPyObjectCtor
Enter
Enter
Exit

Total tests: 182. Passed: 174. Failed: 3. Skipped: 5.
Test Run Failed.
Test execution time: 3.4660 Seconds

henon · 2019-07-03T06:48:21Z

awesome @Esther2013 !
I'll release a MacOS version of Numpy.Bare

lokinfey · 2019-07-03T07:18:26Z

@Esther2013 I have set MONO_OSX , I use this to compile my project

Python.Runtime.csproj.zip

I have followed you to add TRACE;RELEASE;NETSTANDARD;NETSTANDARD2_0;MONO_OSX in the define symbols.

And build it in command line dotnet build -f netstandard2.0 Python.Runtime.csproj

I add Python.Runtime.dll to my project

but I dont't know why it always give me

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0 python3.7m.dylib 0x0000000109de8a37 PyUnicode_InternInPlace + 116
1 python3.7m.dylib 0x0000000109e010b4 PyUnicode_InternFromString + 32
2 python3.7m.dylib 0x0000000109e4c781 PyImport_Import + 87
3 python3.7m.dylib 0x0000000109e4b228 PyImport_ImportModule + 28

Oceania2018 added the bug Something isn't working label Jun 23, 2019

Oceania2018 assigned deepakkumar1984 Jun 23, 2019

Oceania2018 mentioned this issue Jun 27, 2019

Ubuntu env #5

Closed

epignatelli referenced this issue in BHoM/Numpy_Toolkit Nov 21, 2019

Fixes #2

a445637

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

macOS env #2

macOS env #2

lokinfey commented Jun 22, 2019

henon commented Jun 22, 2019

henon commented Jun 22, 2019

lokinfey commented Jun 23, 2019

henon commented Jun 23, 2019

Nucs commented Jun 23, 2019 •

edited

Loading

deepakkumar1984 commented Jun 23, 2019

henon commented Jun 23, 2019

Oceania2018 commented Jun 23, 2019 •

edited

Loading

deepakkumar1984 commented Jun 23, 2019 •

edited

Loading

Nucs commented Jun 24, 2019 •

edited

Loading

deepakkumar1984 commented Jun 24, 2019

Nucs commented Jun 24, 2019 •

edited

Loading

deepakkumar1984 commented Jun 24, 2019

deepakkumar1984 commented Jun 25, 2019

Nucs commented Jun 25, 2019

deepakkumar1984 commented Jun 25, 2019

henon commented Jun 25, 2019

lokinfey commented Jun 30, 2019

henon commented Jun 30, 2019

Oceania2018 commented Jul 1, 2019

Esther2013 commented Jul 3, 2019

henon commented Jul 3, 2019

lokinfey commented Jul 3, 2019 •

edited

Loading

macOS env #2

macOS env #2

Comments

lokinfey commented Jun 22, 2019

henon commented Jun 22, 2019

henon commented Jun 22, 2019

lokinfey commented Jun 23, 2019

henon commented Jun 23, 2019

Nucs commented Jun 23, 2019 • edited Loading

deepakkumar1984 commented Jun 23, 2019

henon commented Jun 23, 2019

Oceania2018 commented Jun 23, 2019 • edited Loading

deepakkumar1984 commented Jun 23, 2019 • edited Loading

Nucs commented Jun 24, 2019 • edited Loading

deepakkumar1984 commented Jun 24, 2019

Nucs commented Jun 24, 2019 • edited Loading

deepakkumar1984 commented Jun 24, 2019

deepakkumar1984 commented Jun 25, 2019

Nucs commented Jun 25, 2019

deepakkumar1984 commented Jun 25, 2019

henon commented Jun 25, 2019

lokinfey commented Jun 30, 2019

henon commented Jun 30, 2019

Oceania2018 commented Jul 1, 2019

Esther2013 commented Jul 3, 2019

henon commented Jul 3, 2019

lokinfey commented Jul 3, 2019 • edited Loading

Nucs commented Jun 23, 2019 •

edited

Loading

Oceania2018 commented Jun 23, 2019 •

edited

Loading

deepakkumar1984 commented Jun 23, 2019 •

edited

Loading

Nucs commented Jun 24, 2019 •

edited

Loading

Nucs commented Jun 24, 2019 •

edited

Loading

lokinfey commented Jul 3, 2019 •

edited

Loading