Reusing `Elixir`: How to Use + Publish Code on Hex.pm 📦

"Code reuse is the Holy Grail of Software Engineering." ~ Douglas Crockford

Once you understand basic Elixir syntax you may be wondering how to reuse both your own code across projects and other people's code in your projects ...

The more (high quality) code you are able to reuse, the more creative and interesting work you can do because you aren't wasting time reimplementing basic functionality or writing boring "boilerplate".

Let's do it!

Reusing Elixir: How to Use + Publish Code on Hex.pm 📦
Why?
What?
- Quotes?
- Problem Statement
How?

Why?

"Good programmers know what to write. Great ones know what to rewrite (and reuse)."
~ Eric S. Raymond (The Cathedral and the Bazaar)

The biggest advantages of code reuse are:

Independently tested small pieces of code that do only one thing. (Curly's Law) 🥇
Work can be subdivided among people/teams with clear responsibilities. ✅
Or if you are solo developer, having small chunks of code helps you bitesize your work so it's more manageable. 🙌
Leverage other people's code to reduce your own efforts and ship faster. 🚀

"If I have seen further than others, it is by standing upon the shoulders of giants." ~ Isaac Newton

We can adapt this quote to a software engineering context as:

"If I have shipped faster and more interesting apps it is by building on the work of giants." ~ Experienced Engineer

What?

In this example we are going to build a simple Elixir module that returns a random inspiring quote.
The functionality of the module is intentionally simple to illustrate code reuse in the most basic form.
Along the way we will demonstrate how to:

Write, document and test a basic package.
Reuse code without publishing to Hex.pm.
Publish a package to Hex.pm
Use the code in a different project.

Quotes?

A quotation, often abbreviated to quote, is the repetition of someone else's statement or thought.
Quotes are usually an expression of wisdom in a concise form. They often condense a lifetime of learning into a single sentence and as such are worthy of our attention.

In our example we will be focussing on a subset of quotes; the thought-provoking kind (often called inspirational or motivational).
e.g:

"If you think you are too small to make a difference, try sleeping with a mosquito." ~ Dalai Lama

"Your time is limited, so don’t waste it living someone else’s life." ~ Steve Jobs

"If you get tired, learn to rest, not to quit." ~ Banksy

There are many uses for quotes. If you're having trouble thinking of how/why this is useful. Imagine a browser home page that displays a different inspiring/motivating/uplifting quote each time you view it to remind you to stay focussed/motivated on your goal for the day.¹

Problem Statement

“First, solve the problem. Then, write the code.” ~ John Johnson

The problem we are solving in this example is: we want to display quotes on our app/website home screen.¹

First we will source some quotes. Then we will create an Elixir module, that when invoked returns a random quote to display.

When Quotes.random() is invoked a map will be returned with the following form:

%{
  "author" => "Peter Drucker",
  "source" => "https://www.goodreads.com/quotes/784267",
  "tags" => "time, management",
  "text" => "Until we can manage time, we can manage nothing else."
}

How?

This is a step-by-step example of creating a reusable Elixir package from scratch.

1. Write Useable Code

"Before software can be reusable, it first has to be usable." ~ Ralph Johnson

1.1 Create a GitHub New Repository

Our first step is always to write useable code. Let's begin by creating a new repository: https://github.com/new

Once you've created the repository, create an issue with the first task. e.g: quotes/issues/1

This makes it clear to yourself (and others) what the next step is.

1.2 Create a New Elixir Project

In a terminal window on your localhost, run the following command:

mix new quotes

That will create all the files needed for our quotes package.
The code created by mix new is: commit/14e7a08 80 additions.

Using the tree command (tree -a lists all files the directory tree and -I '.git' just means "ignore .git directory"):

tree -a -I '.git'

We see that our directory/file structure for the project is:

├── .formatter.exs
├── .gitignore
├── LICENSE
├── README.md
├── lib
│   └── quotes.ex
├── mix.exs
└── test
    ├── quotes_test.exs
    └── test_helper.exs

The interesting/relevant files are these four:

├── lib
│   └── quotes.ex
├── mix.exs
└── test
    ├── quotes_test.exs
    └── test_helper.exs

lib/quotes.ex

defmodule Quotes do
  @moduledoc """
  Documentation for Quotes.
  """

  @doc """
  Hello world.

  ## Examples

      iex> Quotes.hello()
      :world

  """
  def hello do
    :world
  end
end

On creation, the Quotes module has a hello function that returns the :world atom. This is standard in newly created Elixir projects. It will eventually contain our random function.

mix.exs

defmodule Quotes.MixProject do
  use Mix.Project

  def project do
    [
      app: :quotes,
      version: "0.1.0",
      elixir: "~> 1.9",
      start_permanent: Mix.env() == :prod,
      deps: deps()
    ]
  end

  # Run "mix help compile.app" to learn about applications.
  def application do
    [
      extra_applications: [:logger]
    ]
  end

  # Run "mix help deps" to learn about dependencies.
  defp deps do
    [
      # {:dep_from_hexpm, "~> 0.3.0"},
      # {:dep_from_git, git: "https://github.com/elixir-lang/my_dep.git", tag: "0.1.0"}
    ]
  end
end

The mix.exs file is the configuration file for the project/package.

test/quotes_test.exs

defmodule QuotesTest do
  use ExUnit.Case
  doctest Quotes

  test "greets the world" do
    assert Quotes.hello() == :world
  end
end

test/test_helper.exs

ExUnit.start()

Run the Tests

Before writing any code, run the tests to ensure that everything you expect to be working is in fact working:

mix test

You should see something like this:

==> jason
Compiling 8 files (.ex)
Generated jason app
==> quotes
Compiling 2 files (.ex)
Generated quotes app
..

Finished in 0.02 seconds
1 doctest, 1 test, 0 failures

Randomized with seed 771068

That informs us that jason (the dependency we downloaded previously) compiled successfully as did the quotes app.

It also tells us: 1 doctest, 1 test, 0 failures. The doctest (see below) which is the "living documentation" for the hello function executed the example successfully.

Recall that the Example in the @doc block is:

@doc """
Hello world.

## Examples

    iex> Quotes.hello()
    :world

"""
def hello do
  :world
end

If you open iex in your terminal by running iex -S mix
and then input the module and function and run it,
you will see the :world atom as the result:

iex> Quotes.hello()
:world

Doctests are an awesome way of documenting functions because if the function changes the doctest must change with it to avoid breaking.

Update the `hello` function

If we update the hello function to return the atom :kitty instead of :world the doctest will fail.

Try it! Open the lib/quotes.ex file and change the hello function from:

def hello do
  :world
end

To:

def hello do
  :kitty
end

(don't update the @doc/Example yet)

Rerun the tests:

mix test

1) test greets the world (QuotesTest)
   test/quotes_test.exs:5
   Assertion with == failed
   code:  assert Quotes.hello() == :world
   left:  :kitty
   right: :world
   stacktrace:
     test/quotes_test.exs:6: (test)



2) doctest Quotes.hello/0 (1) (QuotesTest)
   test/quotes_test.exs:3
   Doctest failed
   doctest:
     iex> Quotes.hello()
     :world
   code:  Quotes.hello() === :world
   left:  :kitty
   right: :world
   stacktrace:
     lib/quotes.ex:11: Quotes (module)



Finished in 0.03 seconds
1 doctest, 1 test, 2 failures

The doctest failed because the function was updated.

It might seem redundant to have two (similar) tests for the same function. In this simplistic example both the doctest and ExUnit test are testing for the same thing assert Quotes.hello() == :world but the difference is that the doctest example will be included in the module/function's documentation. Always keep in mind that people using your code (including yourself) might not read the tests, but they will rely on the docs so writing doctests are an excellent step.

Change the hello function back to what it was before (returning :world) and let's move on.

Before we can return quotes, we need source a bank of quotes!

Get Quotes!

Now that we have the basics of an Elixir project, our next task is to create (or find) a list of quotes.

We could manually compile our list of quotes by combing through a few popular quotes websites. e.g:

Wikiquote: https://en.wikiquote.org/wiki/Motivation
Brainyquote: https://www.brainyquote.com/topics/motivational-quotes
Goodreads: https://www.goodreads.com/quotes

Or we can feed our favourite search engine with specific targeted keywords. e.g: "inspirational quotes database json free"

Again there are many results so we need to do some sifting ...

https://github.com/public-apis/public-apis
https://opendata.stackexchange.com/questions/3488/large-list-of-quotes
- https://stackoverflow.com/questions/13762688/wiki-quotes-api
https://type.fit/api/quotes
https://github.com/JamesFT/Database-Quotes-JSON
https://github.com/lukePeavey/quotable
https://github.com/skolakoda/programming-quotes-api
https://github.com/jamesseanwright/ron-swanson-quotes

Abracadabra hey presto!

[
  {
    "text": "If I know how you spend your time, then I know what might become of you.",
    "author": "Goethe",
    "source": "https://www.goodreads.com/quotes/6774650",
    "tags": "time, effectiveness"
  },
  {
    "text": "Until we can manage time, we can manage nothing else.",
    "author": "Peter Drucker",
    "source": "https://www.goodreads.com/quotes/784267",
    "tags": "time, management"
  },
  {
    "text": "There is no greater harm than that of time wasted.",
    "author": "Michelangelo",
    "source": "https://www.brainyquote.com/quotes/michelangelo_183580",
    "tags": "time, waste"
  },
  {
    "text": "Those who make the worse use of their time are the first to complain of its shortness",
    "author": "Jean de la Bruyere",
    "source": "https://www.brainyquote.com/quotes/jean_de_la_bruyere_104446",
    "tags": "time, complain"
  },
  {
    "text": "The price of anything is the amount of life you exchange for it.",
    "author": "Henry David Thoreau",
    "source": "https://www.brainyquote.com/quotes/henry_david_thoreau_106427",
    "tags": "price, priorities, life"
  },
  {
    "text": "Life isn't about finding yourself. Life is about creating yourself.",
    "author": "Bernard Shaw",
    "source": "https://www.goodreads.com/quotes/8727",
    "tags": "meaning, creativity"
  },
  {
    "text": "Knowing is not enough, we must apply. Willing is not enough, we must do.",
    "author": "Bruce Lee",
    "source": "https://www.goodreads.com/quotes/302319",
    "tags": "knowledge, action"
  }
]

Full file containing a curated list of quotes: quotes.json

Parsing JSON Data

In order to parse JSON data in Elixir, we need to import a module.

This might seem tedious if you have used other programming languages such as Python or JavaScript which have built-in JSON parsers, but it means we can use a faster parser. And since it all gets compiled down to BEAM bytecode without any effort from the developer, this extra step is automatic.

There are several options to choose from for parsing JSON data on hex.pm (Elixir's package manager)
just search for the keyword "json": https://hex.pm/packages?search=json

In our case we are going to use jason because we have read the code and benchmarks and know that it's good. Phoenix is moving to jason from poison in the next major release.

For a .json file containing only a few thousand quotes it probably does not matter which parser you use. Elixir (or the Erlang VM "BEAM") will cache the decoded JSON map in memory so any of the options will work. Pick one and move on.

Add `jason` to dependencies in `mix.exs`

Open the mix.exs file in your editor and locate the line that starts with

defp deps do

In a new Elixir project the list of deps (dependencies) is empty. Add the following line to the list:

{:jason, "~> 1.1"}

For a snapshot of what the mix.exs file should look like at this point, see: quotes/mix.exs

Run `mix deps.get`

With jason added to the list of deps, you need to run the following command in your terminal:

mix deps.get

This will download the dependency from Hex.pm.

Functions

As always, our first step is to create the user story issue that describes what we are aiming to achieve: quotes/issues/4

The functions we need to create are:

parse_json - open the quotes.json file and parse the contents.
random - get a random quote for any author or topic Quotes.random()
random_by_tag - get a quote by a specific tag e.g: Quotes.tag("time")
random_by_author - get a random quote by a specific author e.g: Quotes.author("Einstein")

Let's start with the first function, opening the quotes.json file and parsing its content.

`Quotes.parse_json`

The functionality for parse_json is quite simple:

open the quotes.json file
parse the data contained in the file
return the parsed data (a List of Maps)

Write the Docs First for the `parse_json` Function

Open the lib/quotes.ex file in your editor and locate the hello function:

def hello do
  :world
end

We are keeping the hello function as reference for writing our own functions for the time being because it's a known state (the tests pass). We will remove it - and the corresponding tests - once the random tests are passing.

Below the hello function, add a new @doc """ block with the following info:

@doc """
parse_json returns a list of maps with quotes in the following form:
[
  %{
    "author" => "Albert Einstein",
    "text" => "Once we accept our limits, we go beyond them."
  },
  %{
    "author" => "Peter Drucker",
    "source" => "https://www.goodreads.com/quotes/784267",
    "tags" => "time, management",
    "text" => "Until we can manage time, we can manage nothing else."
  }
  %{...},
  ...
]

All quotes MUST have an `author` and `text` field.
Some quotes have `tags` and `source`, please help to expand/verify others.
"""

Note on Documentation

The most often overlooked feature in software development is documentation. People naively think that writing the code is all that needs to be done, but that could not be further from the truth. Documentation is at least 30% of the project. Even if you are the only person who will "consume" the reusable code, it still pays to write comprehensive documentation. The relatively small investment pays handsomely when you return to the code in a week/month/year you don't have to waste hours trying to understand it.

"Documentation is a love letter that you write to your future self." ~ Damian Conway

Doctest?

"Incorrect documentation is often worse than no documentation." ~ Bertrand Meyer

Elixir has a superb Doctest feature that helps ensure documentation is kept current. As we saw above, if a function changes and the docs are not updated, the doctests will fail and thus prevent releasing the update.

Given that parse_json returns a large list of maps, it's impractical to add a Doctest example to the @doc block; the doctest would be thousands of lines and would need to be manually updated each time someone adds a quote.

Test `parse_json` Function

Open the test/quotes_test.exs file and add the following code:

test "parse_json returns a list of maps containing quotes" do
  list = Quotes.parse_json()
  assert Enum.count(list) == Utils.count()

  # sample quote we know is in the list
  sample =  %{
    "author" => "Albert Einstein",
    "text" => "A person who never made a mistake never tried anything new."
  }

  # find the sample quote in the List of Maps:
  [found] = Enum.map(list, fn q ->
    if q["author"] == sample["author"] && q["text"] == sample["text"] do
      q
    end
  end)
  |> Enum.filter(& !is_nil(&1)) # filter out any nil values
  assert sample == found # sample quote was found in the list
end

Run the tests in your terminal:

mix test

You should expect to see it fail:

1) test parse_json returns a list of maps containing quotes (QuotesTest)
   test/quotes_test.exs:9
   ** (UndefinedFunctionError) function Quotes.parse_json/0 is undefined or private
   code: list = Quotes.parse_json()
   stacktrace:
     (quotes) Quotes.parse_json()
     test/quotes_test.exs:10: (test)

.

Finished in 0.04 seconds
1 doctest, 2 tests, 1 failure

Make the `parse_json` Test Pass

Add the following code to the lib/quotes.ex file below the @doc definition relevant to the parse_json function

def parse_json do
  File.read!("quotes.json") |> Jason.decode!()
end

Note: For the test to pass, You will also need to create a file called lib/utils.ex and add a count function. See: lib/utils.ex

Re-run the tests:

mix test

You should expect to see the test pass:

...

Finished in 0.06 seconds
1 doctest, 2 tests, 0 failures

Randomized with seed 30116

For good measure, let's write a test that ensures all quotes in quotes.json have an "author" and "text" fields:

test "all quotes have author and text property" do
  Quotes.parse_json()
  |> Enum.each(fn(q) ->
    assert Map.has_key?(q, "author")
    assert Map.has_key?(q, "text")
    assert String.length(q["author"]) > 2 # see: https://git.io/Je8CO
    assert String.length(q["text"]) > 10
  end)
end

This test might seem redundant, but it ensures that people contributing new quotes are not tempted to introduce incomplete data. And having a test that runs on CI, means that the build will fail if quotes are incomplete, which makes the project more reliable.

Document the `Quotes.random()` Function

Now that we have the parse_json helper function, we can move on to the main course!

Open the lib/quotes.ex file (if you don't already have it open), scroll to the bottom and add the following @doc comment:

@doc """
random returns a random quote.
e.g:
[
  %{
    "author" => "Peter Drucker",
    "source" => "https://www.goodreads.com/quotes/784267",
    "tags" => "time, management",
    "text" => "Until we can manage time, we can manage nothing else."
  }
]
"""

Testing `Quotes.random()`

Given that our principal function is random nondeterministic it can be tempting to think that there is "no way to test" it.

In reality it's quite easy to test for randomness, and we can even have a little fun doing it! We currently have 1565 quotes in quotes.json. By running the Quotes.random() function there is a 1 / 1565 x 100 = 0.063% chance of any given quote being returned. That's great because it means people using the quotes will be highly unlikely to see the same quote twice in any given invocation.

But if the person were to keep track of the random quotes they see, the chance of seeing the same quote twice increases with each invocation. This is fairly intuitive, with a finite set of quotes, repetition is inevitable. What is less obvious is how soon the repetition will occur.

Because of a neat feature of compound probability commonly referred to as the "Birthday Paradox", we can calculate exactly when the "random" quotes will be repeated.

We aren't going to dive too deep into probability theory or math, if you are curious about The Birthday Paradox, read Kalid Azad's article (and/or watch his video): https://betterexplained.com/articles/understanding-the-birthday-paradox

Birthday Paradox Formula

We can apply the birthday paradox formula to determine how soon we will see the same "random" quote twice: (replace the word people for quote and days for quotes)

people (number of items we have already seen)     = 200
days (the "population" of available data)         = 1,565
pairs = (people * (people -1)) / 2                = 20,100
chance per pair = pairs / days                    = 12.84345047923
chance different = E^(-chance per pair) * 100     = 0.00026433844
chance of match = (100 - chance different)        = 99.99973566156

There is a 99.9997% probability that at a quote selected at random will match a quote we have already seen after the 200 random events.

In other words if we execute Quotes.random() multiple times and store the result in an List, we are almost certain to see a repeated quote before we reach 200 invocations.

We can translate this into code that tests the Quotes.random function.

Open test/quotes_test.exs file and add the following code:

# This recursive function calls Quotes.random until a quote is repeated
def get_random_quote_until_collision(random_quotes_list) do
  random_quote = Quotes.random()
  if Enum.member?(random_quotes_list, random_quote) do
    random_quotes_list
  else
    get_random_quote_until_collision([random_quote | random_quotes_list])
  end
end

test "Quotes.random returns a random quote" do
  # execute Quotes.random and accumulate until a collision occurs
  random_quotes_list = get_random_quote_until_collision([])
  # this is the birthday paradox at work! ;-)
  # IO.inspect Enum.count(random_quotes_list)
  assert Enum.count(random_quotes_list) < 200
end

If you save the file and run the tests:

mix test

You should expect to see it fail:

1) test Quotes.random returns a random quote (QuotesTest)
   test/quotes_test.exs:49
   ** (UndefinedFunctionError) function Quotes.random/0 is undefined or private
   code: random_quotes_list = get_random_quote_until_collision([])
   stacktrace:
     (quotes) Quotes.random()
     test/quotes_test.exs:41: QuotesTest.get_random_quote_until_collision/1
     test/quotes_test.exs:51: (test)

...

Finished in 0.1 seconds
1 doctest, 4 tests, 1 failure

The test fails because we haven't yet implemented the Quotes.random function.

Let's make the test pass by implementing the function!

Make the `Quotes.random` Test Pass

In the lib/quotes.ex, add the following function definition below the @doc block:

def random do
  parse_json() |> Enum.random()
end

Yep, it's that simple. Elixir is awesome! 🎉

Re-run the tests:

mix test

They should now pass:

.....

Finished in 0.3 seconds
1 doctest, 4 tests, 0 failures

Now that our Quotes.random function is working as expected, we can move on to using the functionality to display quotes.

Tidy Up

Before continuing, take a moment to tidy up the lib/quotes.ex and test/quotes_test.exs files.

Delete the @doc comment and function definition for the hello function. (we no longer need it)
Delete the corresponding test for in the hello function in test/quotes_test.exs Your files should now look like lib/quotes.ex and test/quotes_test.exs

Ensure that the remaining tests still pass (as expected):

mix test

There are fewer tests (because we removed one test and a doctest) but the remaining tests still pass:

Generated quotes app
...

Finished in 0.4 seconds
3 tests, 0 failures

Generate Docs

One of the biggest benefits of writing @doc comments up-front, is that our functions are already documented and we don't have to think about going back and doing it. Elixir can automatically generate the documentation for us!

Add the following line to your mix.exs file in the deps section:

{:ex_doc, "~> 0.21", only: :dev},

Then run the following command to download the ex_doc dependency.

mix deps.get

Now you can run ex_docs with the command:

mix docs

You will see output similar to this:

Compiling 1 file (.ex)
Docs successfully generated.
View them at "doc/index.html".

In your terminal type the following command:

open doc/index.html

That will open the doc/index.html file in your default web browser. e.g:

2. Reuse Code Without Publishing to Hex.pm

3. Publish Package to Hex.pm

What is Hex.pm?

Hex.pm is the package manager for the Elixir (and Erlang) ecosystem. It allows you to publish packages free of charge and share them with your other projects and the community.

Authenticate with hex.pm in the terminal of your localhost:

mix hex.user auth

Publish the package:

mix hex.publish

Building quotes 1.0.0
  Dependencies:
    jason ~> 1.1 (app: jason)
  App: quotes
  Name: quotes
  Files:
    lib
    lib/index.js
    lib/quotes.ex
    lib/utils.ex
    .formatter.exs
    mix.exs
    README.md
    LICENSE
  Version: 1.0.0
  Build tools: mix
  Description: a collection of inspiring quotes and methods to return them.
  Licenses: GNU GPL v2.0
  Links:
    GitHub: https://github.com/dwyl/quotes
  Elixir: ~> 1.9
Before publishing, please read the Code of Conduct: https://hex.pm/policies/codeofconduct

Publishing package to public repository hexpm.

Proceed? [Yn]

Type y in your terminal and hit the [Enter] key.
You will see the following output to confirm the docs have been built:

Building docs...
Compiling 2 files (.ex)
Generated quotes app
Docs successfully generated.
View them at "doc/index.html".

Local password:

Enter the password you defined as part of runnig mix hex.user auth above.

If the package name is available (which we knew it was), then it will be successfully published:

Publishing package...
[#########################] 100%
Package published to https://hex.pm/packages/quotes/1.0.0 (7147b94fa97ee739d8b8a324ed334f7f50566c9ed8632bf07036c31a50bf9c64)
Publishing docs...
[#########################] 100%
Docs published to https://hexdocs.pm/quotes/1.0.0

See: https://hex.pm/packages/quotes

Docs: https://hexdocs.pm/quotes/Quotes.html

4. Use the Package in a New Project

We re-used the quotes package in: https://github.com/dwyl/phoenix-content-negotiation-tutorial

Visit: https://phoenix-content-negotiation.herokuapp.com

You should see a random inspiring quote:

Recap

In this brief tutorial we learned how to write reusable Elixir code. We reused the our Quotes

Modify or Extend?

This example could easily be modified or extended for any purpose.

http://quotes.rest

## Transfer a Package to Another User/Org

https://hex.pm/docs/faq#can-i-transfer-ownership-of-a-package

References and Further Reading

Good background on code reuse: https://en.wikipedia.org/wiki/Code_reuse
Landscape photos: https://unsplash.com/s/photos/landscape
Generating Random Numbers in Erlang and Elixir: https://hashrocket.com/blog/posts/the-adventures-of-generating-random-numbers-in-erlang-and-elixir
TIL: Margaret Hamilton, director of the Software Engineering Division of the MIT Instrumentation Laboratory, which developed on-board flight software for NASA's Apollo space program. (one of the people who coined the term "software engineering") https://en.wikipedia.org/wiki/Margaret_Hamilton_(software_engineer)

Notes

Example Use Case: Momentum Dashboard

Momentum is an example of where inspiring quotes are used. https://momentumdash.com

Though as far as inspiring quotes go, "Yesterday you said tomorrow" is about as inspiring a direct link to the Netflix homepage. 🙄
(we can do so much better than this!)

Even if you feel that having a person homepage/dashboard - that reminds you to stay focussed - is not for you, you can at least acknowledge that there is a huge "market" for it:

Motivational Quotes are Lame

If you are sceptical of motivational quotes, or "self-help" in general, remember that words have motivated many masses.

““Of course motivation is not permanent. But then, neither is bathing;
but it is something you should do on a regular basis.” ~ Zig Ziglar

“I am not young enough to know everything.” ~ Oscar Wilde

You might not think that motivational quotes work on you in the same way that most people feel they aren't influenced to advertising.

Examples of popular quotes (as upvoted or "liked" by the users of goodreads.com): goodreads.com/quotes

Files

code-reuse-hexpm.md

Latest commit

History