Problem building package

[breuderink]

I want to do some experiments with the Atari reinforcement-learning environments. When I tried to install ArcadeLearningEnvironment.jl package I got the following error:

% julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.3 (2020-11-09)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.5) pkg> activate .
 Activating new environment at `/private/tmp/atari/Project.toml`

(atari) pkg> add ArcadeLearningEnvironment
   Updating registry at `~/.julia/registries/General`
  Resolving package versions...
Updating `/private/tmp/atari/Project.toml`
  [b7f77d8d] + ArcadeLearningEnvironment v0.2.3
Updating `/private/tmp/atari/Manifest.toml`
  [b7f77d8d] + ArcadeLearningEnvironment v0.2.3
  [52cbb755] + ArcadeLearningEnvironment_jll v0.6.1+0
  [56f22d72] + Artifacts v1.3.0
  [1fd713ca] + Attr_jll v2.4.48+0
  [6e34b625] + Bzip2_jll v1.0.6+5
  [2e619515] + Expat_jll v2.2.7+6
  [692b3bcd] + JLLWrappers v1.2.0
  [1e303b3e] + LibArchive_jll v3.4.3+1
  [94ce4f54] + Libiconv_jll v1.16.0+7
  [5ced341a] + Lz4_jll v1.9.2+2
  [6ac74813] + MD5 v0.2.1
  [458c3c95] + OpenSSL_jll v1.1.1+6
  [ffd25f8a] + XZ_jll v5.2.4+3
  [83775a58] + Zlib_jll v1.2.11+18
  [3161d3a3] + Zstd_jll v1.4.8+0
  [ed5aba05] + acl_jll v2.2.53+1
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode

(atari) pkg> precompile
Precompiling project...
[ Info: Precompiling ArcadeLearningEnvironment [b7f77d8d-088d-5e02-8ac0-89aab2acc977]
ERROR: LoadError: LoadError: Cannot locate '(Julia)Artifacts.toml' file when attempting to use artifact 'atari_roms' in 'ArcadeLearningEnvironment'
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] @artifact_str(::LineNumberNode, ::Module, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/Artifacts.jl:1052
 [3] include(::Function, ::Module, ::String) at ./Base.jl:380
 [4] include(::Module, ::String) at ./Base.jl:368
 [5] top-level scope at none:2
 [6] eval at ./boot.jl:331 [inlined]
 [7] eval(::Expr) at ./client.jl:467
 [8] top-level scope at ./none:3
in expression starting at /Users/borisreuderink/.julia/packages/ArcadeLearningEnvironment/mdkYD/src/ArcadeLearningEnvironment.jl:9
in expression starting at /Users/borisreuderink/.julia/packages/ArcadeLearningEnvironment/mdkYD/src/ArcadeLearningEnvironment.jl:8

This is on macOS on the M1 architecture. I am new to Julia, so I might do something wrong.

[jbrea]

What happens if you just do using ArcadeLearningEnvironment?

The step pkg> precompile should not be needed. But it shouldn't fail either...

[breuderink]

Thanks for the quick response. I get the same with using ArcadeLearningEnvironment (see below). Please let me know if I can provide more information on this issue.

% julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.3 (2020-11-09)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.5) pkg> activate .
 Activating new environment at `/private/tmp/atari/Project.toml`

(atari) pkg> add ArcadeLearningEnvironment
   Updating registry at `~/.julia/registries/General`
  Resolving package versions...
Updating `/private/tmp/atari/Project.toml`
  [b7f77d8d] + ArcadeLearningEnvironment v0.2.3
Updating `/private/tmp/atari/Manifest.toml`
  [b7f77d8d] + ArcadeLearningEnvironment v0.2.3
  [52cbb755] + ArcadeLearningEnvironment_jll v0.6.1+0
  [56f22d72] + Artifacts v1.3.0
  [1fd713ca] + Attr_jll v2.4.48+0
  [6e34b625] + Bzip2_jll v1.0.6+5
  [2e619515] + Expat_jll v2.2.7+6
  [692b3bcd] + JLLWrappers v1.2.0
  [1e303b3e] + LibArchive_jll v3.4.3+1
  [94ce4f54] + Libiconv_jll v1.16.0+7
  [5ced341a] + Lz4_jll v1.9.2+2
  [6ac74813] + MD5 v0.2.1
  [458c3c95] + OpenSSL_jll v1.1.1+6
  [ffd25f8a] + XZ_jll v5.2.4+3
  [83775a58] + Zlib_jll v1.2.11+18
  [3161d3a3] + Zstd_jll v1.4.8+0
  [ed5aba05] + acl_jll v2.2.53+1
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode

julia> using ArcadeLearningEnvironment
[ Info: Precompiling ArcadeLearningEnvironment [b7f77d8d-088d-5e02-8ac0-89aab2acc977]
ERROR: LoadError: LoadError: Cannot locate '(Julia)Artifacts.toml' file when attempting to use artifact 'atari_roms' in 'ArcadeLearningEnvironment'
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] @artifact_str(::LineNumberNode, ::Module, ::Any) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/Artifacts.jl:1052
 [3] include(::Function, ::Module, ::String) at ./Base.jl:380
 [4] include(::Module, ::String) at ./Base.jl:368
 [5] top-level scope at none:2
 [6] eval at ./boot.jl:331 [inlined]
 [7] eval(::Expr) at ./client.jl:467
 [8] top-level scope at ./none:3
in expression starting at /Users/borisreuderink/.julia/packages/ArcadeLearningEnvironment/mdkYD/src/ArcadeLearningEnvironment.jl:9
in expression starting at /Users/borisreuderink/.julia/packages/ArcadeLearningEnvironment/mdkYD/src/ArcadeLearningEnvironment.jl:8
ERROR: Failed to precompile ArcadeLearningEnvironment [b7f77d8d-088d-5e02-8ac0-89aab2acc977] to /Users/borisreuderink/.julia/compiled/v1.5/ArcadeLearningEnvironment/PemWP_PsfMa.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1305
 [3] _require(::Base.PkgId) at ./loading.jl:1030
 [4] require(::Base.PkgId) at ./loading.jl:928
 [5] require(::Module, ::Symbol) at ./loading.jl:923

julia>

[jbrea]

I wonder if the problem is related to this package or if it is more general. Can you install other packages that rely on artifacts without any problems? For example, does pkg> add FFTW work?

[breuderink]

Yes, I can:

% julia 
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.3 (2020-11-09)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.5) pkg> activate .
 Activating new environment at `/private/tmp/atari/Project.toml`

(atari) pkg> add FFTW
   Updating registry at `~/.julia/registries/General`
######################################################################## 100.0%
  Resolving package versions...
Updating `/private/tmp/atari/Project.toml`
  [7a1cc6ca] + FFTW v1.3.2
Updating `/private/tmp/atari/Manifest.toml`
  [621f4979] + AbstractFFTs v1.0.0
  [56f22d72] + Artifacts v1.3.0
  [7a1cc6ca] + FFTW v1.3.2
  [f5851436] + FFTW_jll v3.3.9+7
  [1d5cc7b8] + IntelOpenMP_jll v2018.0.3+2
  [692b3bcd] + JLLWrappers v1.2.0
  [4af54fe1] + LazyArtifacts v1.3.0
  [856f044c] + MKL_jll v2021.1.1+1
  [189a3867] + Reexport v1.0.0
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode

julia> using FFTW

julia> 

[jbrea]

This is puzzling. I cannot reproduce the error (but I am on Linux) and the closest I found is this and this but these errors don't seem closely related. Could it be that during the download something failed? If so, maybe removing the package pkg> rm ArcadeLearningEnvironment and manually deleting the folders ~/.julia/packages/ArcadeLearningEnvronment_jll and ~/.julia/packages/ArcadeLearningEnvironment and retry pkg> add ArcadeLearningEnvironment could help?

[breuderink]

I just renamed my ~/.julia dir. Now I get a different error; it seems to be related to special characters in the ROM names. If you feel this is unrelated to this repository feel free to close this issue.

% julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.5.3 (2020-11-09)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

(@v1.5) pkg> add ArcadeLearningEnvironment
 Installing known registries into `~/.julia`
######################################################################## 100.0%
      Added registry `General` to `~/.julia/registries/General`
  Resolving package versions...
  Installed ArcadeLearningEnvironment_jll ─ v0.6.1+0
  Installed Bzip2_jll ───────────────────── v1.0.6+5
  Installed XZ_jll ──────────────────────── v5.2.4+3
  Installed OpenSSL_jll ─────────────────── v1.1.1+6
  Installed acl_jll ─────────────────────── v2.2.53+1
  Installed MD5 ─────────────────────────── v0.2.1
  Installed Attr_jll ────────────────────── v2.4.48+0
  Installed ArcadeLearningEnvironment ───── v0.2.3
  Installed Libiconv_jll ────────────────── v1.16.0+7
  Installed Artifacts ───────────────────── v1.3.0
  Installed Lz4_jll ─────────────────────── v1.9.2+2
  Installed Zlib_jll ────────────────────── v1.2.11+18
  Installed Zstd_jll ────────────────────── v1.4.8+0
  Installed Expat_jll ───────────────────── v2.2.7+6
  Installed LibArchive_jll ──────────────── v3.4.3+1
  Installed JLLWrappers ─────────────────── v1.2.0
Downloading artifact: Bzip2
Downloading artifact: ArcadeLearningEnvironment
Downloading artifact: OpenSSL
Downloading artifact: XZ
Downloading artifact: Libiconv
Downloading artifact: Lz4
Downloading artifact: Zlib
Downloading artifact: Zstd
Downloading artifact: LibArchive
Downloading artifact: Expat
Updating `~/.julia/environments/v1.5/Project.toml`
  [b7f77d8d] + ArcadeLearningEnvironment v0.2.3
Updating `~/.julia/environments/v1.5/Manifest.toml`
  [b7f77d8d] + ArcadeLearningEnvironment v0.2.3
  [52cbb755] + ArcadeLearningEnvironment_jll v0.6.1+0
  [56f22d72] + Artifacts v1.3.0
  [1fd713ca] + Attr_jll v2.4.48+0
  [6e34b625] + Bzip2_jll v1.0.6+5
  [2e619515] + Expat_jll v2.2.7+6
  [692b3bcd] + JLLWrappers v1.2.0
  [1e303b3e] + LibArchive_jll v3.4.3+1
  [94ce4f54] + Libiconv_jll v1.16.0+7
  [5ced341a] + Lz4_jll v1.9.2+2
  [6ac74813] + MD5 v0.2.1
  [458c3c95] + OpenSSL_jll v1.1.1+6
  [ffd25f8a] + XZ_jll v5.2.4+3
  [83775a58] + Zlib_jll v1.2.11+18
  [3161d3a3] + Zstd_jll v1.4.8+0
  [ed5aba05] + acl_jll v2.2.53+1
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [b77e0a4c] + InteractiveUtils
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
   Building ArcadeLearningEnvironment → `~/.julia/packages/ArcadeLearningEnvironment/mdkYD/deps/build.log`
┌ Error: Error building `ArcadeLearningEnvironment`: 
│ ROMS/Corrida da Matem?tica (AKA Math Gran Prix) (CCE).bin: Can't create 'ROMS/Corrida da Matem?tica (AKA Math Gran Prix) (CCE).bin'
│ ROMS/Cubo M?gico (AKA Cubicolor) (CCE).bin: Can't create 'ROMS/Cubo M?gico (AKA Cubicolor) (CCE).bin'
│ ROMS/Gopher - Vorsicht W\201hlmaus! (1983) (Carrere Video - JWDA, Sylvia Day, Todd Marshall, Robin McDaniel, Henry Will IV - Teldec - Prism) (USC2001) (PAL).bin: Can't create 'ROMS/Gopher - Vorsicht W\201hlmaus! (1983) (Carrere Video - JWDA, Sylvia Day, Todd Marshall, Robin McDaniel, Henry Will IV - Teldec - Prism) (USC2001) (PAL).bin'
│ ROMS/Lost Luggage - La valise pi\202g\202e (Airport Mayhem) (1982) (Apollo - Games by Apollo, Larry Minor, Ernie Runyon, Ed Salvo - RCA Video Jeux) (AP-2004) (PAL).bin: Can't create 'ROMS/Lost Luggage - La valise pi\202g\202e (Airport Mayhem) (1982) (Apollo - Games by Apollo, Larry Minor, Ernie Runyon, Ed Salvo - RCA Video Jeux) (AP-2004) (PAL).bin'
│ ROMS/Mission 3000 A.D. - Miss?o 3000 A.D. (AKA Mission 3,000 AD) (1983) (CCE) (C-806).bin: Can't create 'ROMS/Mission 3000 A.D. - Miss?o 3000 A.D. (AKA Mission 3,000 AD) (1983) (CCE) (C-806).bin'
│ ROMS/Pega Ladr?o (AKA Keystone Kapers) (Dismac).bin: Can't create 'ROMS/Pega Ladr?o (AKA Keystone Kapers) (Dismac).bin'
│ ROMS/Pel\202's Soccer - Pel\202's Championship Soccer (AKA Championship Soccer) (1981) (Atari, Steve Wright) (CX2616) (PAL).bin: Can't create 'ROMS/Pel\202's Soccer - Pel\202's Championship Soccer (AKA Championship Soccer) (1981) (Atari, Steve Wright) (CX2616) (PAL).bin'
│ ROMS/Pel\202's Soccer - Pel\202's Championship Soccer (AKA Championship Soccer) (1981) (Atari, Steve Wright) (CX2616).bin: Can't create 'ROMS/Pel\202's Soccer - Pel\202's Championship Soccer (AKA Championship Soccer) (1981) (Atari, Steve Wright) (CX2616).bin'
│ ROMS/Space Tunnel - O T?nel Espacial (1983) (CCE) (C-807).bin: Can't create 'ROMS/Space Tunnel - O T?nel Espacial (1983) (CCE) (C-807).bin'
│ ROMS/V?deo Cube (AKA Atari Video Cube) (CCE).bin: Can't create 'ROMS/V?deo Cube (AKA Atari Video Cube) (CCE).bin'
│ bsdtar: Error exit delayed from previous errors.
│ ERROR: LoadError: failed process: Process(`/Users/borisreuderink/.julia/artifacts/ba034a5c3d0363149008455edbdf78ce2f57ab34/bin/bsdtar -xf /Users/borisreuderink/.julia/artifacts/jl_OzYtWK/ROMS.zip`, ProcessExited(1)) [1]
│ 
│ Stacktrace:
│  [1] pipeline_error at ./process.jl:525 [inlined]
│  [2] run(::Cmd; wait::Bool) at ./process.jl:440
│  [3] run at ./process.jl:438 [inlined]
│  [4] (::var"#2#4"{String})(::String) at /Users/borisreuderink/.julia/packages/ArcadeLearningEnvironment/mdkYD/deps/build.jl:42
│  [5] (::LibArchive_jll.var"#14#15"{var"#2#4"{String}})() at /Users/borisreuderink/.julia/packages/LibArchive_jll/4GQAG/src/wrappers/x86_64-apple-darwin14.jl:108
│  [6] withenv(::LibArchive_jll.var"#14#15"{var"#2#4"{String}}, ::Pair{String,String}, ::Vararg{Pair{String,String},N} where N) at ./env.jl:161
│  [7] bsdtar(::var"#2#4"{String}; adjust_PATH::Bool, adjust_LIBPATH::Bool) at /Users/borisreuderink/.julia/packages/LibArchive_jll/4GQAG/src/wrappers/x86_64-apple-darwin14.jl:107
│  [8] bsdtar at /Users/borisreuderink/.julia/packages/LibArchive_jll/4GQAG/src/wrappers/x86_64-apple-darwin14.jl:90 [inlined]
│  [9] (::var"#1#3")(::String) at /Users/borisreuderink/.julia/packages/ArcadeLearningEnvironment/mdkYD/deps/build.jl:40
│  [10] create_artifact(::var"#1#3") at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/Artifacts.jl:214
│  [11] top-level scope at /Users/borisreuderink/.julia/packages/ArcadeLearningEnvironment/mdkYD/deps/build.jl:36
│  [12] include(::String) at ./client.jl:457
│  [13] top-level scope at none:5
│ in expression starting at /Users/borisreuderink/.julia/packages/ArcadeLearningEnvironment/mdkYD/deps/build.jl:35
└ @ Pkg.Operations /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.5/Pkg/src/Operations.jl:949

(@v1.5) pkg> 

[jbrea]

This looks already much better :) Yes, there are some weird characters in the filenames. Usually bsdtar should just ignore them.
I have no clue why libarchives bsdtar doesn't do that on your machine. Maybe your locale is not properly set? Maybe export LC_CTYPE="en_US.UTF-8" in the terminal before opening julia helps?

[breuderink]

I changed my locale, but I get the same error. I'll check if I can run bsdtar manually.

[breuderink]

I tried to manually unpack the ROMs. I get the same error when extracting the ZIP file:

% bsdtar -xf Roms.zip

When I do extract using macOS finder I can extract the content. So the issue appears to be with bsdtar.

[jbrea]

OK. Thanks for checking. Maybe this would be something to report at libarchive? I'll check if there is an alternative to bsdtar for this packages.

After unpacking the files with the maxOS finder you could run the final steps of deps/build.jl manually (basically just import_roms) and then the ROMS should be usable.

[breuderink]

If I can create a concise problem report I'll file at libarchive. Thanks for all the help!

[breuderink]

I did some digging, and I found that I have multiple versions of bsdtar on my system:

1. from macOS, which does not understand .rar files.
2. from Anaconda, which seems to have the issue above.
3. from Homebrew, which doesn't run on the Apple M1 architecture.

While preparing for a bug report for libarchive, I wanted to show that the .rar file is valid. So I tried with p7zip. It does complain that the RAR file is not valid:

borisreuderink@BR-MBA-2021 atari % curl -sO http://www.atarimania.com/roms/Roms.rar             
borisreuderink@BR-MBA-2021 atari % shasum Roms.rar 
00ff84a7e4d6607b4dd57a9aed057eac90b9b394  Roms.rar
borisreuderink@BR-MBA-2021 atari % 7za t Roms.rar

7-Zip (a) [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21
p7zip Version 16.02 (locale=utf8,Utf16=on,HugeFiles=on,64 bits,8 CPUs LE)

Scanning the drive for archives:
1 file, 11128004 bytes (11 MiB)

Testing archive: Roms.rar
ERROR: Roms.rar
Can not open the file as archive

    
Can't open as archive: 1
Files: 0
Size:       0
Compressed: 0

Could the explanation be that the file is not valid, and that different archivers handle the error differently?

A similar warning is produced by bsdtar, but it writes a Roms.zip anyways:

% /opt/homebrew/anaconda3/bin/bsdtar -xf Roms.rar
HC ROMS.zip: Write failed
bsdtar: Error exit delayed from previous errors.
borisreuderink@BR-MBA-2021 atari % shasum Roms.zip
cbb237e3ac8f1a394903de98f364f3ab90dc7122  Roms.zip

[outlace]

I'm getting this same issue on the new Macbook Air with M1 (running Rosetta). Cannot build or use ArcadeLearningEnvironment

[jbrea]

I just updated LibArchive_jll. Can you check if the problem persists? Unfortunately I don't have a Mac at hand to test myself.