Julia crashes with "free(): invalid pointer" or "double free or corruption (out)" on ubuntu 20.04.1 TPU vm #44242
Comments
|
What's the quickest way to access the tpu-vm-pt-1.10? |
A command like:
But I have tested it on v2-alpha with same result |
|
Same error on TPU-VM here. |
|
@jekbradbury, do you have any insight about what may be happening here? |
|
Sorry, do we have an update on this? Thanks. |
|
Unfortunately no. Could someone produce stack traces with a Julia nightly and Julia 1.8-rc1 available at https://julialang.org/downloads/ ? |
|
Same error: With nightly: If there's a more useful stack trace I can produce let me know. |
|
Thanks. Can we isolate the error to the following line?
julia/stdlib/LibGit2/src/LibGit2.jl Line 986 in 63c6983 In other words, is executing the following line sufficient to crash Julia? ccall((:git_libgit2_init, :libgit2), Cint, ()) |
|
The next step would be to verify that |
|
The current binary build for LibGit2 included Julia is grabbed from https://github.com/JuliaBinaryWrappers/LibGit2_jll.jl The build script here is here: |
|
As far as I can tell, none of the bundled patches that Julia applies touches code in the stack trace. https://github.com/JuliaPackaging/Yggdrasil/tree/master/L/LibGit2/bundled/patches |
|
Is |
|
Nothing with tpu-vm-tf-2.8.0 : With tpu-vm-pt-1.11: |
Could you try this? |
|
For the sake of making the package manager usable, you can set the environment variable Side note, in case you wanted to try again to compile Julia with |
|
With preview ccall does produce same stacktrace. Unfortunately setting JULIA_PKG_USE_CLI_GIT=true does not seem to do anything. |
Did you export the variable? What's the value of |
|
Yes, I exported the variable. I also tried both true and 1. |
|
|
|
What does the trace look like for the conda-forge build? That build uses libgit2 built against openssl rather than mbedtls so the stack trace should not use Otherwise, I would try to build libgit2 directly and see if their tests are passing. Maybe @giordano knows how to set this up with our bundled patches applied? |
|
@mkitti If you just want to build git clone https://github.com/libgit2/libgit2.git
# get patch
wget https://raw.githubusercontent.com/JuliaPackaging/Yggdrasil/master/L/LibGit2/bundled/patches/libgit2-agent-nonfatal.patch
wget https://raw.githubusercontent.com/JuliaPackaging/Yggdrasil/master/L/LibGit2/bundled/patches/libgit2-hostkey.patch
wget https://raw.githubusercontent.com/JuliaPackaging/Yggdrasil/master/L/LibGit2/bundled/patches/libgit2-win32-ownership.patch
# checkout version
cd libgit2/
git checkout v1.4.3
# apply patch
patch -p1 -f < ../libgit2-agent-nonfatal.patch
patch -p1 -f < ../libgit2-hostkey.patch
patch -p1 -f < ../libgit2-win32-ownership.patch
# build flags
LIBGIT2_BUILD_FLAGS="-DCMAKE_BUILD_TYPE=Release -DUSE_THREADS=ON -DUSE_BUNDLED_ZLIB=ON -DUSE_SHA1=CollisionDetection"
# open tests and examples
LIBGIT2_BUILD_FLAGS="$LIBGIT2_BUILD_FLAGS -DBUILD_TESTS=ON -DBUILD_EXAMPLES=ON"
# use OpenSSL
LIBGIT2_BUILD_FLAGS="$LIBGIT2_BUILD_FLAGS -DUSE_HTTPS=\"OpenSSL\""
# build
mkdir build && cd build
cmake .. "$LIBGIT2_BUILD_FLAGS"
make -j `nproc`
|
|
Mentioning this also here: it's known that various Google environments (Google Cloud Platform, Google Colab, etc...) may set |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm trying to run julia on a tpu-vm v3-8 using the tpu-vm-pt-1.10 image. It crashes on various operations with "free(): invalid pointer." This happens with 1.7.2 binary, the 1.6.5 LTS, conda-forge version and a similar error occurs when building from source or using conda-forge.
For example
Machine info:
I did notice that trying to check the libc.so.6 info segfaulted, which makes me suspicious google is doing something strange with their glibc.
I tried to make julia, and got a related crash:
I found a very similar issue in discourse, but it had no replies. Github issues did not seem to have anything relevant. https://discourse.julialang.org/t/issues-with-julia-installation-on-google-tpu-vm/65783
I posted and got advice about various flags to try. I tried various combinations without success, though its not clear if the failure was for the same reason.
Output:
default_options.txt
USE_BINARYBUILDER=0.txt
USE_BINARYBUILDER=0-USE_BINARY_BUILDER_LIBGIT2=0.txt
USE_BINARYBUILDER=0-USE_SYSTEM_CURL=1.txt
USE_SYSTEM_CURL=1.txt
Those using USE_BINARYBUILDER=0 failed with
Has anyone had success with julia on TPU vms? Thanks!
The text was updated successfully, but these errors were encountered: