-
Notifications
You must be signed in to change notification settings - Fork 370
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run jstack
against Spark Driver process failed on MacOS M1
#5702
Comments
jstack
failed on Spark Driver process running on MacOS M1jstack
against Spark Driver process failed on MacOS M1
Verified that:
|
Adding JVM option
|
According to this doc, I ran the following test cases and observed different behavior of .java_pid file.
|
A simplified program call reproduce this Bug. import java.io.File;
import java.io.FileOutputStream;
import java.io.InputStream;
public class AttachListener {
public static void main(String[] args) throws InterruptedException {
File file = extractVeloxLibrary();
System.load(file.getAbsolutePath());
System.out.println("Library velox loaded");
Thread.sleep(Long.MAX_VALUE);
}
static File extractVeloxLibrary() {
String tmpdir = System.getProperty("java.io.tmpdir");
File file = new File(tmpdir, "libvelox.dylib");
if (file.exists()) {
file.delete();
}
try (InputStream is = AttachListener.class.getResourceAsStream("/libvelox.dylib");
FileOutputStream fos = new FileOutputStream(file)) {
byte[] buffer = new byte[4096];
int read;
while ((read = is.read(buffer)) != -1) {
fos.write(buffer, 0, read);
}
} catch (java.io.IOException e) {
throw new RuntimeException("Failed to extract library", e);
}
return file;
}
} Compile and run: javac AttachListener.java
java -cp /path/to/incubator-gluten/package/target/gluten-velox-bundle-spark3.5_2.12-osx_14.4_aarch_64-1.2.0-SNAPSHOT.jar:/path/to/spark-3.5.1-bin-hadoop3/jars/*:. AttachListener
|
I guess libvelox.dylib affected JVM's internal mechanism. OpenJDK Project has a similar issue: https://bugs.openjdk.org/browse/JDK-8235211, but seems not relevant. |
I am using macOS(Apple Silicon), JDK:
and jstack works |
Backend
VL (Velox)
Bug description
Launch spark-sql in local mode and run
jstack
against it:jstack
exits with error messageSpark version
spark-3.5.1-bin-hadoop3
Spark configurations
No response
System information
JDK
System
Relevant logs
No response
The text was updated successfully, but these errors were encountered: