Deploying with releases

[ghost]

Elixir 1.9's releases are a built-in replacement for Distillery. Like Distillery, you can't run Mix tasks against a release.

I tried following the deployment guide in the docs but run into an error:

Starting dependencies...
Starting repos...
Starting clusters...
Building indexes...
** (exit) exited in: GenServer.call(Backend.Elasticsearch.Cluster, :config, 5000)
    ** (EXIT) no process: the process is not alive or there's no process currently associated with the given name, possibly because its application isn't started
    (elixir) lib/gen_server.ex:1000: GenServer.call/3
    (elasticsearch) lib/elasticsearch/indexing/index.ex:32: Elasticsearch.Index.hot_swap/2
    (elixir) lib/enum.ex:783: Enum."-each/2-lists^foreach/1-0-"/2
    (elixir) lib/enum.ex:783: Enum.each/2
    lib/backend/release.ex:29: Backend.Release.build_elasticsearch_indexes/0
    (stdlib) erl_eval.erl:680: :erl_eval.do_apply/6
    (elixir) lib/code.ex:240: Code.eval_string/3

This is the release module that I'm running:

defmodule Backend.Release do
  @app :backend
  @start_apps [
    :crypto,
    :ssl,
    :postgrex,
    :ecto,
    :elasticsearch
  ]

  # Ecto repos to start, if any
  @repos Application.get_env(:backend, :ecto_repos, [])
  # Elasticsearch clusters to start
  @clusters [Backend.Elasticsearch.Cluster]
  # Elasticsearch indexes to build
  @indexes [:instances]

  def build_elasticsearch_indexes() do
    start_services()
    IO.puts("Building indexes...")
    Enum.each(@indexes, &Elasticsearch.Index.hot_swap(Backend.Elasticsearch.Cluster, &1))
    stop_services()
  end

  # Ensure that all OTP apps, repos used by your Elasticsearch store,
  # and your Elasticsearch Cluster(s) are started
  defp start_services do
    IO.puts("Starting dependencies...")
    Enum.each(@start_apps, &Application.ensure_all_started/1)
    IO.puts("Starting repos...")
    Enum.each(@repos, & &1.start_link(pool_size: 1))
    IO.puts("Starting clusters...")
    Enum.each(@clusters, & &1.start_link())
  end

  defp stop_services do
    :init.stop()
  end
end

If I replace the contents of start_services() with a single line that calls Application.ensure_all_started(@app) then things work fine, but this starts my entire app which I'd prefer to avoid.

Does anyone know if there's a major difference between Distillery and Elixir releases that could be causing this? It seems like Enum.each(@clusters, & &1.start_link()) is not starting the cluster as it should.

[ghost]

Some additional details here:

With Application.ensure_all_started(@app), the indexing runs, but attempts to read from the index fail with a no such index error. No indexes are actually created by the hot_swap task.

[danielberkompas]

@brortao You are going to need to add your app's name to the list of apps to start, since the Backend.Elasticsearch.Cluster process is part of your app and won't start otherwise. If you prefer, you could possibly spawn it as part of this task, but I'm not sure exactly what that would look like.

Regarding the no such index error, are you able to get hot_swap to run and create an index when you run it locally?

[ghost]

Hmm, yes, running the task locally does create the index. It's just on my deployed service. I'm guessing this is a problem with my infrastructure so I'll close this and continue there!

[ghost]

Ah, hang on, reopening. It's failing because it cannot find the index config JSON file. That's why it was working on my local machine: when I run it from my project folder, it works; when I run it from elsewhere I get an enoent error.

I think the problem is in hot_swap when it calls create_from_file, which calls File.read.

Here are the results of File.ls on the deployed app:

iex(backend@5714fba14b09)10> File.ls()                          
{:ok, ["lib", "tmp", "bin", "erts-10.4.4", "releases", "Procfile"]}

The index configuration is hidden in a subfolder:

iex(backend@5714fba14b09)12> File.ls("./lib/backend-2.2.0/priv/elasticsearch")
{:ok, ["instances.json"]}

I've tried to create an index from this file, but there isn't a neat solution:

iex(backend@5714fba14b09)6> Elasticsearch.Index.create_from_file(Backend.Elasticsearch.Cluster, "instances-1", "instances.json")                                     
{:error, :enoent}

iex(backend@5714fba14b09)4> Elasticsearch.Index.create_from_file(Backend.Elasticsearch.Cluster, "instances-1", "elasticsearch/instances.json")
{:error, :enoent}

iex(backend@5714fba14b09)5> Elasticsearch.Index.create_from_file(Backend.Elasticsearch.Cluster, "instances-1", "lib/backend-2.2.0/priv/elasticsearch/instances.json")
:ok

[ghost]

I opened a thread on the Elixir forums to find out if there's a good way to access files in priv/ from a Mix release: https://elixirforum.com/t/accessing-the-priv-folder-from-mix-release/24403

[semarco]

@brortao you can simply use :code.priv_dir(:your-app) to get the absolute priv/path of your OTP application at runtime. There is also Application.app_dir(:your-app) and Application.app_dir(:your-app, "relative_path/foo/bar").

It would really help to simplify deployments if it would be possible to put index-settings.json files into the priv folder. :-)
Thanks for looking into!!

[ghost]

@pixelvitamina yeah, those work at runtime, but as far as I can tell this library needs the path during configuration at compile-time. You can't use those functions in e.g config.exs.

Since you can't pass the settings file directory to hot_swap, I think it may need to be included there?

elasticsearch-elixir/lib/elasticsearch/indexing/index.ex

Line 33 in 1a7bbe7

%{settings: settings_file} = index_config = config[:indexes][alias]

[ImmaculatePine]

@brortao You can add a runtime configuration by overriding init/1 callback in your cluster module: https://github.com/danielberkompas/elasticsearch-elixir/blob/master/lib/elasticsearch/cluster/cluster.ex#L37

[ghost]

Amazing, I didn't realize this. Thank you!

[millacas]

Arrived a little late to the conversation. If anyone is interested this is what I used to make the indexes work on a release:

defmodule MyApp.ElasticsearchCluster do
  use Elasticsearch.Cluster, otp_app: :my_app
  @app :my_app

  def init(config) do
    indexes =
      config[:indexes]

    indexes =
      Map.keys(indexes)
      |> Enum.reduce(indexes, fn index_key, indexes ->
        Map.update!(indexes, index_key, fn index ->
          Map.update!(index, :settings, fn settings_file_path ->
            Application.app_dir(@app, settings_file_path)
          end)
        end)
      end)

    config =
      Map.put(config, :indexes, indexes)

    {:ok, config}
  end
end