Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with install on mac M2 - import py5 in Jupyter causes kernel to die #316

Closed
buchholzmd opened this issue Jul 12, 2023 · 28 comments
Closed

Comments

@buchholzmd
Copy link

I followed these install steps but was unable to install Java 17 (or 20) using install-jdk due to a 404 error, so had to use openjdk.

Upon starting the py5 kernel, it immediately dies. When using the created conda env as the kernel, the kernel dies after importing py5.

I am using a mac M2. Jupyter gives the following error:

2023-07-12 16:09:38.875 python[4306:86435] *** Assertion failure in +[NSEvent otherEventWithType:location:modifierFlags:timestamp:windowNumber:context:subtype:data1:data2:], NSEvent.m:647
2023-07-12 16:09:38.884 python[4306:86435] *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Invalid parameter not satisfying: _NSEventMask64FromType(type) & WeirdMask'
*** First throw call stack:
(
	0   CoreFoundation                      0x000000019009b3f8 __exceptionPreprocess + 176
	1   libobjc.A.dylib                     0x000000018fbe6ea8 objc_exception_throw + 60
	2   Foundation                          0x0000000190fdca60 -[NSCalendarDate initWithCoder:] + 0
	3   AppKit                              0x0000000193444250 +[NSEvent otherEventWithType:location:modifierFlags:timestamp:windowNumber:context:subtype:data1:data2:] + 264
	4   libffi.8.dylib                      0x0000000104bcc04c ffi_call_SYSV + 76
	5   libffi.8.dylib                      0x0000000104bc974c ffi_call_int + 1208
	6   _ctypes.cpython-39-darwin.so        0x0000000104bac544 _ctypes_callproc + 1324
	7   _ctypes.cpython-39-darwin.so        0x0000000104ba6850 PyCFuncPtr_call + 1176
	8   python3.9                           0x00000001043bd3c4 _PyObject_MakeTpCall + 616
	9   python3.9                           0x00000001044ad464 call_function + 668
	10  python3.9                           0x00000001044a9c48 _PyEval_EvalFrameDefault + 26456
	11  python3.9                           0x00000001043be0a0 function_code_fastcall + 116
	12  python3.9                           0x00000001044ad3cc call_function + 516
	13  python3.9                           0x00000001044a9c48 _PyEval_EvalFrameDefault + 26456
	14  python3.9                           0x00000001044a2e98 _PyEval_EvalCode + 2804
	15  python3.9                           0x00000001043bdfe4 _PyFunction_Vectorcall + 220
	16  _ctypes.cpython-39-darwin.so        0x0000000104baa920 closure_fcn + 644
	17  libffi.8.dylib                      0x0000000104bc9b34 ffi_closure_SYSV_inner + 808
	18  libffi.8.dylib                      0x0000000104bcc1d4 ffi_closure_SYSV + 52
	19  CoreFoundation                      0x000000019003a55c __CFRUNLOOP_IS_CALLING_OUT_TO_A_TIMER_CALLBACK_FUNCTION__ + 32
	20  CoreFoundation                      0x000000019003a204 __CFRunLoopDoTimer + 940
	21  CoreFoundation                      0x0000000190039d5c __CFRunLoopDoTimers + 356
	22  CoreFoundation                      0x000000019001f744 __CFRunLoopRun + 1896
	23  CoreFoundation                      0x000000019001e888 CFRunLoopRunSpecific + 612
	24  HIToolbox                           0x00000001996f3fa0 RunCurrentEventLoopInMode + 292
	25  HIToolbox                           0x00000001996f3de4 ReceiveNextEventCommon + 672
	26  HIToolbox                           0x00000001996f3b2c _BlockUntilNextEventMatchingListInModeWithFilter + 72
	27  AppKit                              0x00000001932a0424 _DPSNextEvent + 632
	28  AppKit                              0x000000019329f5b4 -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 728
	29  AppKit                              0x00000001932939e4 -[NSApplication run] + 464
	30  libffi.8.dylib                      0x0000000104bcc04c ffi_call_SYSV + 76
	31  libffi.8.dylib                      0x0000000104bc974c ffi_call_int + 1208
	32  _ctypes.cpython-39-darwin.so        0x0000000104bac544 _ctypes_callproc + 1324
	33  _ctypes.cpython-39-darwin.so        0x0000000104ba6850 PyCFuncPtr_call + 1176
	34  python3.9                           0x00000001043bd3c4 _PyObject_MakeTpCall + 616
	35  python3.9                           0x00000001044ad464 call_function + 668
	36  python3.9                           0x00000001044a9c48 _PyEval_EvalFrameDefault + 26456
	37  python3.9                           0x00000001044a2e98 _PyEval_EvalCode + 2804
	38  python3.9                           0x00000001043bdfe4 _PyFunction_Vectorcall + 220
	39  python3.9                           0x00000001044ad3cc call_function + 516
	40  python3.9                           0x00000001044a9c48 _PyEval_EvalFrameDefault + 26456
	41  python3.9                           0x00000001044a2e98 _PyEval_EvalCode + 2804
	42  python3.9                           0x00000001043bdfe4 _PyFunction_Vectorcall + 220
	43  python3.9                           0x00000001044ad3cc call_function + 516
	44  python3.9                           0x00000001044a9c48 _PyEval_EvalFrameDefault + 26456
	45  python3.9                           0x00000001044a2e98 _PyEval_EvalCode + 2804
	46  python3.9                           0x00000001043bdfe4 _PyFunction_Vectorcall + 220
	47  python3.9                           0x0000000104555740 partial_vectorcall + 396
	48  python3.9                           0x00000001044ad3cc call_function + 516
	49  python3.9                           0x00000001044a9c48 _PyEval_EvalFrameDefault + 26456
	50  python3.9                           0x00000001043be0a0 function_code_fastcall + 116
	51  python3.9                           0x00000001043c0f24 method_vectorcall + 516
	52  python3.9                           0x00000001044c18c0 context_run + 348
	53  python3.9                           0x000000010440d050 cfunction_vectorcall_FASTCALL_KEYWORDS + 136
	54  python3.9                           0x00000001044a9ef4 _PyEval_EvalFrameDefault + 27140
	55  python3.9                           0x00000001043be0a0 function_code_fastcall + 116
	56  python3.9                           0x00000001044ad3cc call_function + 516
	57  python3.9                           0x00000001044a9ba8 _PyEval_EvalFrameDefault + 26296
	58  python3.9                           0x00000001043be0a0 function_code_fastcall + 116
	59  python3.9                           0x00000001044ad3cc call_function + 516
	60  python3.9                           0x00000001044a9ba8 _PyEval_EvalFrameDefault + 26296
	61  python3.9                           0x00000001043be0a0 function_code_fastcall + 116
	62  python3.9                           0x00000001044ad3cc call_function + 516
	63  python3.9                           0x00000001044a9ba8 _PyEval_EvalFrameDefault + 26296
	64  python3.9                           0x00000001043be0a0 function_code_fastcall + 116
	65  python3.9                           0x00000001044ad3cc call_function + 516
	66  python3.9                           0x00000001044a9ba8 _PyEval_EvalFrameDefault + 26296
	67  python3.9                           0x00000001043be0a0 function_code_fastcall + 116
	68  python3.9                           0x00000001044ad3cc call_function + 516
	69  python3.9                           0x00000001044a9ba8 _PyEval_EvalFrameDefault + 26296
	70  python3.9                           0x00000001044a2e98 _PyEval_EvalCode + 2804
	71  python3.9                           0x00000001043bdfe4 _PyFunction_Vectorcall + 220
	72  python3.9                           0x00000001043c0dc4 method_vectorcall + 164
	73  python3.9                           0x00000001044ad3cc call_function + 516
	74  python3.9                           0x00000001044a9bcc _PyEval_EvalFrameDefault + 26332
	75  python3.9                           0x00000001044a2e98 _PyEval_EvalCode + 2804
	76  python3.9                           0x00000001044a2398 PyEval_EvalCode + 76
	77  python3.9                           0x000000010449df40 builtin_exec + 844
	78  python3.9                           0x000000010440d198 cfunction_vectorcall_FASTCALL + 216
	79  python3.9                           0x00000001044ad3cc call_function + 516
	80  python3.9                           0x00000001044a9c48 _PyEval_EvalFrameDefault + 26456
	81  python3.9                           0x00000001044a2e98 _PyEval_EvalCode + 2804
	82  python3.9                           0x00000001043bdfe4 _PyFunction_Vectorcall + 220
	83  python3.9                           0x00000001044ad3cc call_function + 516
	84  python3.9                           0x00000001044a9c48 _PyEval_EvalFrameDefault + 26456
	85  python3.9                           0x00000001044a2e98 _PyEval_EvalCode + 2804
	86  python3.9                           0x00000001043bdfe4 _PyFunction_Vectorcall + 220
	87  python3.9                           0x000000010451cf68 pymain_run_module + 272
	88  python3.9                           0x000000010451c6a8 Py_RunMain + 1612
	89  python3.9                           0x000000010451da40 pymain_main + 1252
	90  python3.9                           0x0000000104370770 main + 56
	91  dyld                                0x000000018fc17e50 start + 2544
)
libc++abi: terminating with uncaught exception of type NSException
[I 2023-07-12 16:09:40.553 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), new random ports
@hx2A
Copy link
Collaborator

hx2A commented Jul 12, 2023

Welcome, @buchholzmd ! Let's see if we can figure this out.

Upon starting the py5 kernel, it immediately dies. When using the created conda env as the kernel, the kernel dies after importing py5.

Starting the py5 kernel imports py5 for you, so let's focus on why the kernel dies when you import py5. And that's a pretty horrific error message you have there. There must be a problem of some kind with how the JDK was installed.

Can you provide the output to py5_tools.get_jvm_debug_info()? Do the JAVA_HOME and default jvm path values make sense to you for your setup?

Here's what I get on my Mac computer:

% ipython
import Python 3.8.16 | packaged by conda-forge | (default, Feb  1 2023, 16:05:36) 
Type 'copyright', 'credits' or 'license' for more information
IPython 8.12.2 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import py5_tools
py
In [2]: py5_tools.get_jvm_debug_info()
Out[2]: 
{'JAVA_HOME environment variable': '/usr/local/Cellar/openjdk/17.0.2/',
 'jvm version': (0, 0, 0),
 'default jvm path': '/usr/local/Cellar/openjdk/17.0.2/libexec/openjdk.jdk/Contents/Home/lib/libjli.dylib'}

@buchholzmd
Copy link
Author

buchholzmd commented Jul 12, 2023

Thanks for a quick response! To be clear - I did not install Java on my computer, but only within the conda environment (not sure if that matters though).

After running the debugging method I get the following:
Screenshot 2023-07-12 at 4 58 42 PM

After installing Java on my computer (as well as installing it in the conda env using openjdk), I now get a new (and less ugly) error message:


#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x000000018fbd5808, pid=5308, tid=259
#
# JRE version: OpenJDK Runtime Environment Zulu20.28+85-CA (20.0+36) (build 20+36)
# Java VM: OpenJDK 64-Bit Server VM Zulu20.28+85-CA (20+36, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, bsd-aarch64)
# Problematic frame:
# C  [libobjc.A.dylib+0x9808]  objc_msgSend+0x8
#
# No core dump will be written. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /Users/matthewbuchholz/Documents/Art/genart/hs_err_pid5308.log
#
# If you would like to submit a bug report, please visit:
#   http://www.azul.com/support/
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

@hx2A
Copy link
Collaborator

hx2A commented Jul 12, 2023

was unable to install Java 17 (or 20) using install-jdk due to a 404 error

The install-jdk library supports installing various JDKs and JREs builds. You can specify a different vendor besides the default Adoptium. You can also get the download URL it is attempting to use for debugging purposes:

In [6]: import jdk

In [7]: jdk.get_download_url('17')
Out[7]: 'https://api.adoptium.net/v3/binary/latest/17/ga/mac/x64/jdk/hotspot/normal/eclipse'

In [8]: jdk.get_download_url('17', vendor='Azul')
Out[8]: 'https://cdn.azul.com/zulu/bin/zulu17.42.19-ca-jdk17.0.7-macosx_x64.tar.gz'

Both of those URLs work for me right now. But if the default option gives you a 404 error, you can always try a different one.

I suspect there is something amiss with how the JDK was installed on your computer. I know that install-jdk does it correctly.

@hx2A
Copy link
Collaborator

hx2A commented Jul 12, 2023

Thanks for a quick response! To be clear - I did not install Java on my computer, but only within the conda environment (not sure if that matters though).

That's it! Please don't install Java with anaconda. The JREs/JDKs that it will install are problematic. I've seen weird dependencies where installing matplotlib will downgrade your Java version, which makes no sense to me. I suggest starting with a new Anaconda environment and installing Java with install-jdk, perhaps using a different vendor.

@buchholzmd
Copy link
Author

I believe you are correct that this is an issue with my JDK install... unfortunately, I cannot resolve the issue yet.

So far, I have removed Java and JDK from my mac and started from scratch. I was able to get install-jdk to work by trying another vendor, as mentioned. First issue that arises is a permissions error from trying to run jdk.install when not the root user, so the only way I can install using install-jdk is with the following command:

sudo python -c "import jdk; print('Java installed to', jdk.install('17', vendor='Azul'))"

After this step, when running java --version in the terminal, I receive the message stating that java is not installed. The only fix to this is downloading the Java .dmg from Oracle's website directly. However, now the output of the py5_tools method is the following:

Screenshot 2023-07-13 at 10 02 49 PM

I think my problem boils down to configuring the correct directory for $JAVA_HOME, but I have yet to find the correct setup and am further confused that the default jvm path is now /Library/Java/JavaVirtualMachines/ (which does not work for $JAVA_HOME).

@hx2A
Copy link
Collaborator

hx2A commented Jul 14, 2023

First issue that arises is a permissions error from trying to run jdk.install when not the root user, so the only way I can install using install-jdk is with the following command:

Hmmm, something's wrong here. You shouldn't need sudo permission because it will by default install it in $HOME/.jdk/<VERSION> or $HOME/.jre/<VERSION>. Is there something funny with your $HOME variable? If not, can you try opening an issue for this with for that library's repo at https://github.com/jyksnw/install-jdk? I've corresponded with the maintainer of that project before, he is diligent about getting back to people. Feel free to @ me in the issue you create there if you would like.

I think my problem boils down to configuring the correct directory for $JAVA_HOME, but I have yet to find the correct setup and am further confused that the default jvm path is now /Library/Java/JavaVirtualMachines/ (which does not work for $JAVA_HOME).

But before opening an issue, can you try setting your JAVA_HOME to /Library/Java/JavaVirtualMachines/jdk-17.jdk? And if that doesn't work, try each subdirectory navigating towards the bin subdirectory that contains the java executable?

@buchholzmd
Copy link
Author

Interesting, so a few things:

  1. $HOME is set to /Users/myname
  2. Trying to set JAVA_HOME to /Library/Java/JavaVirtualMachines/jdk-17.jdk yields zsh: /Library/Java/JavaVirtualMachines/jdk-17.jdk/ not found ... the same goes for trying /opt/homebrew/Cellar/openjdk@17/17.0.7/. I have confirmed that both directories do in fact exist... sigh
  3. Thanks for your willingness to help. This isn't a py5 issue as much as it's user error ... I've used py5 in the past without issue and just excited to get back into it again

@hx2A
Copy link
Collaborator

hx2A commented Jul 14, 2023

After setting JAVA_HOME, what do you see when you type echo $JAVA_HOME? In the Python kernel, what happens when you look in os.environ['JAVA_HOME']?

In [1]: import os

In [2]: os.environ['JAVA_HOME']
Out[2]: '/usr/lib/jvm/java-17-openjdk'

(My machine here is Linux w/ Java installed by the OS in /usr/lib/jvm)

@buchholzmd
Copy link
Author

I tried setting JAVA_HOME to be /Library/Java/JavaVirtualMachines/jdk-17.jdk/, /opt/homebrew/Cellar/openjdk@17/17.0.7/ and /opt/homebrew/Cellar/openjdk@17/. Each time os.environ['JAVA_HOME'] reports the correct, corresponding values. Still kernel is dying upon importing py5

@hx2A
Copy link
Collaborator

hx2A commented Jul 14, 2023

Hmmm, I don't understand what the issue is. Can you work on getting install-jdk to install without sudo? That, so far, has proven to be a reliable way for others to install Java on OSX. Where is install-jdk trying to install to anyway? Do you have some permission issue where you can't create new directories in your home directory?

@buchholzmd
Copy link
Author

Yes, I started looking around a bit at past issues for install-jdk but with no luck. I'll open an issue like you suggested

It's trying to install in $HOME/.jdk/<VERSION>, so yes, I am having permissions denied in my home directory for some reason

@hx2A
Copy link
Collaborator

hx2A commented Jul 14, 2023

Can you create any directories in your home directory? $HOME/temp/? If not, you should fix that first. It has something to do with ACLs, I believe. Is this a new computer? I vaguely remember doing something to address this when I got a mac a few years ago.

@buchholzmd
Copy link
Author

@hx2A Yes, this is a new Mac, so this must be the case. What is weird is that I did grant permissions for folders under $HOME, and yes I can mkdir from $HOME, so there is clearly some finer-grain permissions needed that I need to sort out.

Thanks for the help, once I figure out how to run this not as root, I reckon things will be good to go!

@buchholzmd
Copy link
Author

Some progress: for some reason, when using install-jdk the directories it creates are owned by root rather than user. Not sure if this is an issue with how conda/miniforge is configured on my machine or my mac in general but will update once that's solved

@hx2A
Copy link
Collaborator

hx2A commented Jul 14, 2023

It's not Ananconda. Previously you said you first ran install-jdk with sudo, which is why root owns that directory. Can you try removing that directory completely and running install-jdk without sudo? If that doesn't work, figure out what the problem is without using sudo. I don't see why you need sudo rights if you can use mkdir in $HOME. Some confusing Mac thing.

@buchholzmd
Copy link
Author

buchholzmd commented Jul 16, 2023

Ok, thank you. The problem was that I forgot to remove the full jdk directory. Now I can run install-jdk without sudo. After doing that java --version still says the Java runtime cannot be found, but setting JAVA_HOME to be $HOME/.jdk/zulu17.42.19-ca-jdk17.0.7-macosx_aarch64/zulu-17.jdk/Contents/Home, Java can find the OpenJDK runtime. But unfortunately the kernel again dies upon importing py5, with the same nasty NSException

@hx2A
Copy link
Collaborator

hx2A commented Jul 16, 2023

Can you try this in a regular Python interpreter:

>>> import jpype
>>> jpype.startJVM()

?

If this doesn't work, it must be a jpype problem, an issue with the JVM, or an issue your machine. In that case, I would try other JVMs options provided by install-jdk. Can you try with the default Adoptium?

If that does work, it a py5 problem. I'm running out of ideas for what to do though.

@buchholzmd
Copy link
Author

So starting JVM with jpype does work. This is all with, Azul btw.

Adoptium was giving the 404 error with install-jdk. I can try to download Adoptium either with brew or from a webpage and just setting JAVA_HOME based on that install maybe?

@buchholzmd
Copy link
Author

Including the log for reference

hs_err_pid1326.log

@hx2A
Copy link
Collaborator

hx2A commented Jul 16, 2023

Can you double check you have the latest version of install-jdk?

$ pip install install-jdk --upgrade
Requirement already satisfied: install-jdk in /home/jim/INSTALL/anaconda3/envs/py5/lib/python3.8/site-packages (1.0.4)

It should be 1.0.4.

Next, here is the download URL for adoptium Java 17:

>>> jdk.get_download_url('17')
'https://api.adoptium.net/v3/binary/latest/17/ga/linux/x64/jdk/hotspot/normal/eclipse'

Does that URL work for you in a browser? Is install-jdk still giving a 404 error? That should have been a temporary thing. Do you get a different download URL than me?

You can also try installing Java with the appropriate DMG installer from here:

https://www.oracle.com/java/technologies/downloads/#jdk17-mac

That should be really easy to get working. Then remove or adjust $JAVA_HOME for the new location. If you need to figure out where it installed Java to, try which java in a terminal.

@buchholzmd
Copy link
Author

buchholzmd commented Jul 16, 2023

Yes, I have 1.0.4, and btw I'm using python 3.9.

The url you provided (for linux) does work, but for mac arm it's not working: running the get_url method it yields
https://api.adoptium.net/v3/binary/latest/17/ga/mac/arm/jdk/hotspot/normal/eclipse

However, I changed to this architecture and this url works:
https://api.adoptium.net/v3/binary/latest/17/ga/mac/aarch64/jdk/hotspot/normal/eclipse

Still receive the same error of kernel dying.

And btw, I initially tried the DMG install from Oracle's website. Just tried again with JAVA_HOME set to the same path as which java (/usr/bin/java/) as well as the actual Home directory installed (/Library/Java/JavaVirtualMachines/jdk-17.jdk/Contents/Home/). Still kernel dies with same stack trace as in the log earlier

@hx2A
Copy link
Collaborator

hx2A commented Jul 16, 2023

Maybe this is an architecture issue? Perhaps you get a 404 error because Adoptium does not have a build for the architecture install-jdk detects is correct for your machine? You can't install a build for a different architecture. This could explain why it is crashing the way that it is.

@buchholzmd
Copy link
Author

Possibly... but what confuses me is that aarch64 is the correct architecture. And further, from my understanding aarch64 and arm64 are equivalent. So not sure what the difference between arm and aarch64 is here (on the install-jdk pypi page they list all the architectures that can be detected).

I've opened an issue with install-jdk and hopefully can get some clarification.

@hx2A
Copy link
Collaborator

hx2A commented Jul 16, 2023

The different architectures are confusing to me, especially on the mac computers. Let's see what the install-jdk maintainer has to say.

Side note: are you at NYU Courant? I graduated from there in 2005 with a MS in Math & Finance.

@buchholzmd
Copy link
Author

Agreed - will keep this thread updated! And thanks for all the help btw

And yes I am! I actually checked out your blog a few days ago and saw you were an alum. I watched the video of your ITP thesis and found it really cool! I actually worked with spherical convolutions a few times before and your work would have provided some useful references xD

@hx2A
Copy link
Collaborator

hx2A commented Jul 19, 2023

Any updates? Are you still trying to get py5 working on your machine?

@hx2A
Copy link
Collaborator

hx2A commented Aug 7, 2023

@buchholzmd Is this complete? Can I close this issue?

@hx2A
Copy link
Collaborator

hx2A commented Dec 26, 2023

I am going to close this. Please open a new issue for further mac problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants