Elixir-based tests fail to run with Elixir 1.15

[pgj]

For the latest version of CouchDB main (as the time of creating this ticket), it is not possible run the Elixir-based integration tests with Elixir 1.15.

Consider the following log where Elixir is 1.15.4 is used (via asdf, built for OTP 25, and CouchDB is also built with that version):

% make elixir-suite
* creating /Users/gaborpali/.asdf/installs/elixir/1.15.4-otp-25/.mix/elixir/1-15/rebar3
* creating /Users/gaborpali/.asdf/installs/elixir/1.15.4-otp-25/.mix/archives/hex-2.0.6
Resolving Hex dependencies...
Resolution completed in 0.029s
Unchanged:
  bunt 0.2.1
  certifi 2.9.0
  credo 1.6.7
  excoveralls 0.15.3
  file_system 0.2.10
  hackney 1.18.1
  httpotion 3.2.0 RETIRED!
    (deprecated) Not really maintained, please check out Tesla
  idna 6.1.1
  jason 1.4.0
  junit_formatter 3.3.1
  metrics 1.0.1
  mimerl 1.2.0
  parse_trans 3.3.1
  ssl_verify_fun 1.1.6
  unicode_util_compat 0.7.0
===> Analyzing applications...
===> Compiling b64url
===> Analyzing applications...
===> Compiling jwtf
===> Analyzing applications...
===> Compiling jiffy
===> Analyzing applications...
===> Compiling ibrowse
==> httpotion
Compiling 1 file (.ex)

18:28:45.831 [error] beam/beam_load.c(144): Error loading module ibrowse:
  corrupt file header



18:28:45.831 [error] Loading of /Users/gaborpali/projects/github/couchdb/_build/integration/lib/ibrowse/ebin/ibrowse.beam failed: :badfile

warning: :ibrowse.send_req/6 is undefined (module :ibrowse is not available or is yet to be defined)
  lib/httpotion.ex:476: HTTPotion.request/3

warning: :ibrowse.send_req_direct/7 is undefined (module :ibrowse is not available or is yet to be defined)
  lib/httpotion.ex:476: HTTPotion.request/3

warning: :ibrowse.stop_worker_process/1 is undefined (module :ibrowse is not available or is yet to be defined)
  lib/httpotion.ex:476: HTTPotion.stop_worker_process/1

warning: :ssl_verify_hostname.verify_fun/3 is undefined (module :ssl_verify_hostname is not available or is yet to be defined)
  lib/httpotion.ex:476: HTTPotion.process_arguments/3

Generated httpotion app
==> couchdbtest
warning: :meck.unload/0 is undefined (module :meck is not available or is yet to be defined)
  test/elixir/lib/ex_unit.ex:39

warning: :config.get/3 is undefined (module :config is not available or is yet to be defined)
  test/elixir/lib/step/start.ex:43

warning: :config.get/3 is undefined (module :config is not available or is yet to be defined)
  test/elixir/lib/step/start.ex:49

warning: :mochiweb_socket_server.get/2 is undefined (module :mochiweb_socket_server is not available or is yet to be defined)
  test/elixir/lib/step/start.ex:44

warning: :mochiweb_socket_server.get/2 is undefined (module :mochiweb_socket_server is not available or is yet to be defined)
  test/elixir/lib/step/start.ex:50

warning: :test_util.start_couch/1 is undefined (module :test_util is not available or is yet to be defined)
  test/elixir/lib/step/start.ex:42

warning: :test_util.stop_couch/1 is undefined (module :test_util is not available or is yet to be defined)
  test/elixir/lib/step/start.ex:63

warning: :fabric.create_db/2 is undefined (module :fabric is not available or is yet to be defined)
  test/elixir/lib/step/create_db.ex:39

warning: :fabric.delete_db/2 is undefined (module :fabric is not available or is yet to be defined)
  test/elixir/lib/step/create_db.ex:45

warning: :ibrowse.send_req/6 is undefined (module :ibrowse is not available or is yet to be defined)
  test/elixir/lib/couch_raw.ex:2

warning: :ibrowse.send_req_direct/7 is undefined (module :ibrowse is not available or is yet to be defined)
  test/elixir/lib/couch_raw.ex:2

warning: :ibrowse.stop_worker_process/1 is undefined (module :ibrowse is not available or is yet to be defined)
  test/elixir/lib/couch_raw.ex:2

warning: :jiffy.encode/1 is undefined (module :jiffy is not available or is yet to be defined)
  test/elixir/lib/couch_raw.ex:49

warning: :ssl_verify_hostname.verify_fun/3 is undefined (module :ssl_verify_hostname is not available or is yet to be defined)
  test/elixir/lib/couch_raw.ex:2

warning: :config.delete/3 is undefined (module :config is not available or is yet to be defined)
  test/elixir/lib/step/user.ex:79

warning: :config.get/3 is undefined (module :config is not available or is yet to be defined)
  test/elixir/lib/step/user.ex:36

warning: :config.set/4 is undefined (module :config is not available or is yet to be defined)
  test/elixir/lib/step/user.ex:48

warning: :couch_doc.from_json_obj/1 is undefined (module :couch_doc is not available or is yet to be defined)
  test/elixir/lib/step/user.ex:59

warning: :couch_doc.from_json_obj/1 is undefined (module :couch_doc is not available or is yet to be defined)
  test/elixir/lib/step/user.ex:83

warning: :fabric.create_db/2 is undefined (module :fabric is not available or is yet to be defined)
  test/elixir/lib/step/user.ex:41

warning: :fabric.delete_db/2 is undefined (module :fabric is not available or is yet to be defined)
  test/elixir/lib/step/user.ex:39

warning: :fabric.get_doc_info/3 is undefined (module :fabric is not available or is yet to be defined)
  test/elixir/lib/step/user.ex:82

warning: :fabric.update_doc/3 is undefined (module :fabric is not available or is yet to be defined)
  test/elixir/lib/step/user.ex:66

warning: :fabric.update_doc/3 is undefined (module :fabric is not available or is yet to be defined)
  test/elixir/lib/step/user.ex:88

warning: :ibrowse.send_req/6 is undefined (module :ibrowse is not available or is yet to be defined)
  test/elixir/lib/couch.ex:69

warning: :ibrowse.send_req_direct/7 is undefined (module :ibrowse is not available or is yet to be defined)
  test/elixir/lib/couch.ex:69

warning: :ibrowse.stop_worker_process/1 is undefined (module :ibrowse is not available or is yet to be defined)
  test/elixir/lib/couch.ex:69

warning: :jiffy.decode/2 is undefined (module :jiffy is not available or is yet to be defined)
  test/elixir/lib/couch.ex:134

warning: :jiffy.encode/1 is undefined (module :jiffy is not available or is yet to be defined)
  test/elixir/lib/couch.ex:120

warning: :ssl_verify_hostname.verify_fun/3 is undefined (module :ssl_verify_hostname is not available or is yet to be defined)
  test/elixir/lib/couch.ex:69

warning: :jiffy.encode/1 is undefined (module :jiffy is not available or is yet to be defined)
  test/elixir/lib/couch/dbtest.ex:93

warning: :jiffy.encode/1 is undefined (module :jiffy is not available or is yet to be defined)
  test/elixir/lib/couch/dbtest.ex:112

warning: :jiffy.encode/1 is undefined (module :jiffy is not available or is yet to be defined)
  test/elixir/lib/couch/dbtest.ex:260

warning: :jiffy.encode/1 is undefined (module :jiffy is not available or is yet to be defined)
  test/elixir/lib/couch/dbtest.ex:476

warning: :jiffy.encode/1 is undefined (module :jiffy is not available or is yet to be defined)
  test/elixir/lib/couch/dbtest.ex:507

warning: :fabric.get_db_info/1 is undefined (module :fabric is not available or is yet to be defined)
  test/elixir/lib/utils.ex:26

Generated couchdbtest app

18:28:45.914 [error] Loading of /Users/gaborpali/projects/github/couchdb/_build/integration/lib/ibrowse/ebin/ibrowse_app.beam failed: :badfile


18:28:45.914 [error] beam/beam_load.c(144): Error loading module ibrowse_app:
  corrupt file header



18:28:45.914 [notice] Application ibrowse exited: exited in: :ibrowse_app.start(:normal, [])
    ** (EXIT) an exception was raised:
        ** (UndefinedFunctionError) function :ibrowse_app.start/2 is undefined (module :ibrowse_app is not available)
            (ibrowse 4.4.2) :ibrowse_app.start(:normal, [])
            (kernel 8.5.4.1) application_master.erl:293: :application_master.start_it_old/4
** (Mix) Could not start application ibrowse: exited in: :ibrowse_app.start(:normal, [])
    ** (EXIT) an exception was raised:
        ** (UndefinedFunctionError) function :ibrowse_app.start/2 is undefined (module :ibrowse_app is not available)
            (ibrowse 4.4.2) :ibrowse_app.start(:normal, [])
            (kernel 8.5.4.1) application_master.erl:293: :application_master.start_it_old/4
make: *** [elixir-suite] Error 1

The size of _build/integration/lib/ibrowse/ebin/ibrowse_app.beam is 0, which may explain the first error message. Also, for further invocations, the command simply hangs.

[jjrodrig]

Hi @pgj!

I've been checking this issue.
Initially I had the same error related to ibrowse. I fully cleaned up the elixir environment and then a new issue appears with the jiffy module.

Could you try if it is the same for you?

1 - Clean CouchDB build
make clean
2 - Clean up elixir build artifacts

 mix clean --deps
 rm src/*/_build
 rm _build

3 - Rebuild CouchDB and test

 make couch elixir

I get this kind of errors

 1) test Basic reduce functions (ReduceTest)
     test/elixir/test/reduce_test.exs:16
     ** (UndefinedFunctionError) function :jiffy.decode/2 is undefined (module :jiffy is not available)
     stacktrace:
       (jiffy 1.0.9) :jiffy.decode("{\"ok\":true}\n", [:return_maps])

If it is the case, I think the problem is related to this change (https://hexdocs.pm/elixir/1.15.4/changelog.html#potential-incompatibilities)

By adding prune_code_paths: false to the project section in mix.exs file solves the issue for me, around line 49 on the file.

  def project do
    [
      app: :couchdbtest,
      version: "0.1.0",
      elixir: "~> 1.13",
      lockfile: Path.expand("mix.lock", __DIR__),
      deps_path: Path.expand("src", __DIR__),
      build_path: Path.expand("_build", __DIR__),
      compilers: [:elixir, :app],
      start_permanent: Mix.env() == :prod,
      build_embedded: Mix.env() == :prod,
      deps: deps(),
      consolidate_protocols: Mix.env() not in [:test, :dev, :integration],
      test_paths: get_test_paths(Mix.env()),
      elixirc_paths: elixirc_paths(Mix.env()),
      prune_code_paths: false,
      test_coverage: [
        tool: CoverTool,
        dirs: get_coverage_paths(),
        type: "html"
      ]
    ]
  end

Seems to be a temporary solution but unlock the build using Elixir 1.15. I'll prepare a PR with this change.

[pgj]

Hi @jjrodrig : Yes, this issue sometimes happens with jiffy as well. I have experimented with the prune_code_paths compiler flag too, but it looks like I put that to a wrong place because I could not see any improvements. But it is good that I was proven to be wrong and I am happy to test your PR.

[nickva]

Wonder if there is a way to declare a proper dependency on jiffy if we're actually using it.

[jjrodrig]

Jiffy is used for encoding/decoding requests/responses during testing. It is a core element in the elixir test suite.

I was checking what is wrong with this dependency but I din't found anything.

Jiffy is properly declared as a local elixir dependency in mix.exs file.

defp deps() do
    deps_list = [
      {:junit_formatter, "~> 3.0", only: [:dev, :test, :integration]},
      {:httpotion, ">= 3.1.3", only: [:dev, :test, :integration], runtime: false},
      {:excoveralls, "~> 0.12", only: :test},
      {:b64url, path: path("b64url")},
      {:jiffy, path: path("jiffy")},
      {:jwtf, path: path("jwtf")},
      {:ibrowse, path: path("ibrowse"), override: true},
      {:credo, "~> 1.6.4", only: [:dev, :test, :integration], runtime: false}
    ]
 ...

MIX_ENV=integration mix deps responds that dependency is ok

 jiffy 1.0.9 (/Users/juanjo/projects/couchdb/src/jiffy) (rebar3)
  ok

Elixir 1.15 release notes indicates this potential incompatibility.

Due to the code path pruning, if you have an application or dependency that does not specify its dependencies on Erlang and Elixir application, it may no longer compile successfully in Elixir v1.15. You can temporarily disable code path pruning by setting prune_code_paths: false in your mix.exs, although doing so may lead to runtime bugs that are only manifested inside a mix release

In our case, if we set prune_code_paths: false tests works, but if we unset this property tests compile but fail to load jiffy at runtime. Any clue about what to check will be apreciated.

[pgj]

jiffy uses a couple of NIFs -- maybe that is related? I have faced similar problems when I tried to use jiffy in a project where I wanted to do rebar3 escriptize.

[nickva]

Thanks for fixing Elixir 1.15! It's nice to be up-to-date.

One downside I think is it broke the deb package building in the full (main) CI. I noticed it in the PR: #4726 (comment)

It seems to complain about missing hex and trying to install it to build junit_formatter. Wonder if it's because we removed _build or ran clean --deps while before we relied on packaging those in the dist tarball (perhaps by accident).

[jjrodrig]

Hi @nickva

It seems that the packaging process depends on the elixir environment but it is not prepared as we cleaned it completely. Some step is missing to recover it but i don't know where.

As a quick fix I can separate the elixir clean from the main clean target. So the new clean steps will not interfere in the current CI.

[nickva]

Thanks @jjrodrig, yeah maybe a separate clean step would work.

To verify it would pass or not a pr branch that starts with jenkins- prefix would end up running both the quick PR CI and the full main one. That can help with checking if it would work or not without having to figure out the couchdb-pkg locally.