ChoreRunner

An Elixir library for writing and running code chores.

• Motivation
• Installation
• Writing a Chore
• Chore Reporting
• Telemetry
• Contribution

A "Chore" can really be anything, but most commonly it is just some infrequently, manually run code which achieve a business or development goal.

For example: updating a config value in a database that does not yet have a UI (perhaps due to time constraints) is a great use for a chore. A chore could be created that accepts the desired value and runs the update query.

The primary fast alternative to extensive development for some infrequent business need tends to be a direct prod-shell or prod-db connection, which is inherently insecure and dangerous. Many fast-moving startups or companies are ok with this access for developers, and that's fine. But many companies have regulations that they must follow, or do not want to take the risk of a developer mistake while working in these environments.

ChoreRunner allows the rapid creation, testing, and reviewing of code chores, along with a bundled UI for running them that accepts a variety of input types, with the goal of finding a "sweet spot" of safety and speed when solving such problems.

• Remote shells skip testing and review processes that could detect issues with the code being executed. Chores are easily tested and reviewed, and tested/reviewed code is safer than untested and unreviewed code.
• Remote-shell access is a common security audit issue. To comply with security requirements such as PCI-DSS, they might need to be restricted. Chores are a normal part of your app, with controlled behavior like any other admin page, and will not flag most security audits (depending on how you use them).
• Two developers could unwittingly step on each other's toes if using remote shells in the same app simultaneously. Chores have built-in protections for this.
• A common reason for using shells is the lack of developer bandwidth to create specific admin functionality for an uncommon task. ChoreRunner ships with a LiveView UI that generates input forms automatically, and lets you track chore progress.

Add chore_runner to your deps.

{:chore_runner, "~> 0.1.3"}

Add ChoreRunner to your supervision tree, after your app's PubSub:

children = [
  {Phoenix.PubSub, [name: MyApp.PubSub]},
  {ChoreRunner, [pubsub: MyApp.PubSub]},
]

Decide your app's chore namespace and maybe write an example chore:

defmodule MyApp.Chores.MyFirstChore do
  use ChoreRunner.Chore
  # Chore namespace is MyApp.Chores
  def run(_), do: :ok
end

Add the UI to your router, passing otp_app, chore_root, and pubsub keys in the session:

@chore_session %{"otp_app" => :my_app, "chore_root" => MyApp.Chores, "pubsub" => MyApp.PubSub}
scope "/" do
  pipe_through :browser

  live_session :chores, session: @chore_session do
    live "/chores", ChoreRunnerUI.ChoreLive, :index
    # Optional, use if you want to use the downloads feature
    live "/chores/downloads", ChoreRunnerUI.DownloadsLive, :index
  end
end

Your Phoenix app must be LiveView-enabled for the UI to work properly.

You can now visit your UI at the route specified.

The most basic chore module looks like this

defmodule MyApp.Chores.BasicChore do
  use ChoreRunner.Chore

  def run(_) do
  # ...
  end
end

Besides run/1, chores can implement optional callbacks for inputs, cross-field validation, instructions, availability, concurrency restrictions, result handling, and URL persistence.

defmodule MyApp.Chores.BasicChore do
  use ChoreRunner.Chore

  def inputs do
    [
      string(:my_string),
      int(:my_int),
      float(:my_float),
      bool(:my_bool),
      file(:my_file),
    ]
  end

  def run(_inputs) do
  # ...
  end
end

The inputs callback lets you define expected inputs to the chore using input functions imported from ChoreRunner.Input. Input not defined in the callback will be discarded when passed to a chore through ChoreRunner.run_chore/2. The 6 supported input types are:

• string
• int
• float
• bool
• file
• selectbox

Select inputs accept either Phoenix-compatible options or a zero-arity function that returns them:

def inputs do
  [
    selectbox(:role, [{"Admin", :admin}, {"User", :user}], default: :user),
    selectbox(:workspace, &workspace_options/0, prompt: "Choose a workspace")
  ]
end

Submitted select values are checked against the current options and converted back to the value type declared by the option.

An input can depend on inputs declared before it with :depends_on. A dependent input stays hidden, and is not validated, until every input it depends on is valid. A select options provider with one argument receives the validated values of those inputs, so options can be loaded for the current choice:

def inputs do
  [
    selectbox(:region, &region_options/0, prompt: "Choose a region"),
    selectbox(:city, &city_options/1, depends_on: [:region], prompt: "Choose a city"),
    selectbox(:district, &district_options/1, depends_on: [:city], prompt: "Choose a district"),
    selectbox(:mode, [{"Summary", "summary"}, {"Detailed", "detailed"}], default: "summary"),
    int(:guests, depends_on: [:mode], visible?: &(&1.mode == "detailed"))
  ]
end

defp city_options(%{region: region}), do: MyApp.Places.city_options(region)

• Submitted values are validated against the options loaded for the submitted dependency values, both in the UI and in ChoreRunner.run_chore/3.
• :visible? receives the same dependency values and hides the input when it returns anything other than true.
• Hidden inputs are left out of the map passed to validate_inputs/1 and run/1, and are not persisted in the URL.
• When an input changes in the UI, its dependents return to their prompt or default value.
• Dependencies must be declared earlier in the list and cannot be file inputs. Chores without :depends_on behave exactly as before.

Input functions also accept an optional keyword list of options as a second argument. The supported keys are:

• :description — a description of an input
• :default — a value used when the caller omits the input and prefilled in the UI
• :validators — a list of functions, either captured or anonymous, that can be used to transform and validate any provided input.

  def inputs do
    [
      string(:name, validators: [&check_length/1, & {:ok, String.capitalize(&1)}]),
    ]
  end
  
  defp check_length(string) do
    if String.length(string) < 3 do
      {:error, "Must be 3 or more characters in lenth"}
    else
      :ok
    end
  end

  Each input type has a default validator that are always run, which do basic type validation.

Use validate_inputs/1 when a validation rule depends on more than one input. It runs only after defaults, type casting, and every individual field validator have succeeded, and receives the complete atom-keyed input map.

Return {:ok, validated_inputs} to continue, or {:error, keyword_errors} to reject the input. Each error key must name a declared input and each value must be a non-empty list of reasons. The bundled UI displays these errors beside the referenced fields and disables the Run button.

def validate_inputs(%{buyer_type: :registered, number_of_orders: count})
    when rem(count, 2) != 0 do
  {:error, number_of_orders: ["must be even for registered buyers"]}
end

def validate_inputs(inputs), do: {:ok, inputs}

Chores that do not implement this callback use a default pass-through implementation.

Chores can opt in to persisting all non-file inputs in the URL. This makes a configured form reloadable and shareable. Because URLs may be stored in browser history and application logs, only enable this for chores whose inputs are safe to expose.

use ChoreRunner.Chore, persist_inputs_in_url?: true

Persisted values use the inputs[...] query namespace. File inputs are always excluded, and values are validated normally before a chore can run.

The chore-name filter is always stored in the non-sensitive filter query parameter and restored on reload.

Return a string to show in a collapsed Instructions panel above the chore inputs:

def instructions do
  "Run the dry-run mode first, review the output, then enable commit."
end

When instructions/0 is not defined or returns nil, the UI falls back to the chore module's @moduledoc. The panel is omitted when neither is available.

The restriction callback configures each chore's concurrency restriction. Concurrency restriction potentially prevents a chore from running depending on what chores are currently running. This includes chores running on other nodes. It supports 3 valid return values:

• def restriction, do: :none

  The chore has no restrictions. Multiple of this chore can be run similtaneously.

• def restriction, do: :self

  Only one of this specific chore is allowed to run at a time.

• def restriction, do: :global

  Only one :global chore can run at a time, even between different chores. This does not prevent other non-global chores from running.

The meat of your chore will reside in the run/1 callback. When you run a chore through the UI, that calls ChoreRunner.run_chore/2, which eventually calls out to your run/1 callback. It accepts an atom-keyed map of expected input, defined by the input callback. Input presence in the map is not garaunteed, but you are garaunteed to only receive input specified in the callback. Any return value is is stored in the chore struct via the reporter, broadcasted on the pubsub, and also directly passed to the configured chore resolution handler on chore completion.

use ChoreRunner.Chore imports the following functions for use in the run/1 callback:

• log(message)

  Logs a timestamped string

• set_counter(counter_key, counter_value)

  Sets the specified counter to the provided value

• inc_counter(counter_key, value_to_increment_by)

  Increments the specified counter by the provided value. If the value does not exist, the value defaults to 0, then is incremented.

These functions work in both the main chore process, and certain spawned processes such as via Task.async_stream for parellelization. Attempting to call these functions outside of those conditions will result in an exception.

use ChoreRunner.Chore also imports the function create_download(filename, opts). This allows your chores to save files which can be downloaded right after they are created, either via a link on the chore progress element on ChoreLive liveview page, or from the dedicated DownloadsLive liveview page.

To use, simply provide a filename and either a path to a file, or a file body. Filenames do not need to be unique, and chores know which files they created automatically (Though a file does not know its parent chore).

# Local file path
create_download("mytextfile.txt", path: "path/to/file.txt")

# Binary body
create_download("mytextfile.txt", body: "this is my text file!")

The download is saved and loaded using the configured storage service.

The default storage service is ChoreRunner.Downloads.TemporaryDiskStorageService, an ephemeral local disk solution that is multi-node compatible out of the box!

You may implement your own solution—for example, uploading files to a cloud platform—by implementing the callbacks specified in the ChoreRunner.Downloads.StorageService behaviour. You can then configure the global storage_service in your application config.

config :chore_runner, :storage_service, MyStorageService

The following telemetry events are sent from the ChoreRunner.Reporter

• [:chore_runner, :reporter, :chore_failed]
  • Emitted when chore has failed.
  • Sends %{state: chore_state, error_reason: reason}.
• [:chore_runner, :reporter, :chore_finished]
  • Emitted when chore has finished.
  • Sends %Chore{result: nil} where the result is set to nil to avoid sending large results through telemetry.
• [:chore_runner, :reporter, :init]
  • Emitted when the reporter is started.
  • Sends %{chore: chore, opts: init_opts}.
• [:chore_runner, :reporter, :log]
  • Emitted when a log is added.
  • Sends %{state: chore_state}.
• [:chore_runner, :reporter, :start_chore]
  • Emitted when a chore is started.
  • Sends %{state: chore_state}.
• [:chore_runner, :reporter, :stop_chore]
  • Emitted when a chore is stopped.
  • Sends %{status: status, state: chore_state} where state is :ok | :error.
• [:chore_runner, :reporter, :update_counter]
  • Emitted when counter is updated.
  • Sends %{state: chore_state}.

Unfortunately, we cannot accept pull requests at this time. However, we will adress any and all issues opened related to bugs or ideas/features.