10-fine-tune-trading-strategy.Rmd

# Fine-tune trading strategy per symbol

## Objectives

- describe and design the required functionality
- add docker to project
- set up `ecto` inside the `naive` app
- create and migrate the DB
- seed symbols' settings
- update the `Naive.Leader` to fetch settings

## Describe and design the required functionality

At this moment the settings of our naive strategy are hardcoded inside the `Naive.Leader`:

```{r, engine = 'elixir', eval = FALSE}
  # /apps/naive/lib/naive/leader.ex
  ...
  defp fetch_symbol_settings(symbol) do
    symbol_filters = fetch_symbol_filters(symbol)

    Map.merge(
      %{
        symbol: symbol,              # <=
        chunks: 5,                   # <=
        budget: 100,                 # <=
        # -0.01% for quick testing   # <=
        buy_down_interval: "0.0001", # <= all of those settings
        # -0.12% for quick testing   # <=
        profit_interval: "-0.0012",  # <=
        rebuy_interval: "0.001"      # <=
      },
      symbol_filters
    )
  end
  ...
```

The problem about those is that they are hardcoded and there's no flexibility to define them per symbol at the moment.

In this chapter, we will move them out from this file into the Postgres database.

## Add docker to project

The requirements for this section are `docker` and `docker-compose` installed in your system.

Inside the main directory of our project create a new file called `docker-compose.yml` and fill it with the below details:

```{r, engine = 'yaml', eval = FALSE}
# /docker-compose.yml
version: "3.2"
services:
  db:
    image: postgres:latest
    restart: always
    environment:
      POSTGRES_PASSWORD: "hedgehogSecretPassword"
    ports:
      - 5432:5432
    volumes:
      - ../postgres-data:/var/lib/postgresql/data
```

If you are new to docker here's the gist of what the above will do:

- it will start a single service called "db"
- "db" service will use the `latest` version of the `postgres` (image) inside the docker container (`latest` version as tagged per https://hub.docker.com/_/postgres/)
- we map TCP port 5432 in the container to port 5432 on the Docker host(format container_port:hosts_port)
- we set up environmental variable inside the docker container that will be used by the Postgres app as a password for the default (`postgres`) user
- `volumes` option maps the directory from inside of the container to the host. This way we will keep the state of the database between restarts.

\newpage

We can now start the service using `docker-compose`:

```{r, engine = 'bash', eval = FALSE}
$ docker-compose up -d
Creating hedgehog_db_1 ... done
```

To validate that it works we can run:

```{r, engine = 'bash', eval = FALSE}
$ docker ps -a
CONTAINER ID   IMAGE             COMMAND                  CREATED         STATUS
PORTS                    NAMES
98558827b80b   postgres:latest   "docker-entrypoint.sh"   4 seconds ago   Up 4 seconds
0.0.0.0:5432->5432/tcp   hedgehog_db_1
```



## Set up `ecto` inside the `naive` app

Let's start by adding database access to the `naive` application. The first step is to add the [Ecto](https://github.com/elixir-ecto/ecto) module together with the [Postgrex](https://github.com/elixir-ecto/postgrex) ecto's driver package to the `deps` function inside the `mix.exs` file. As we are going to use Enums inside Postgres, we need to add the [EctoEnum](https://github.com/gjaldon/ecto_enum) module as well:

```{r, engine = 'elixir', eval = FALSE}
  # /apps/naive/mix.exs
  defp deps do
    [
      {:binance, "~> 1.0"},
      {:binance_mock, in_umbrella: true},
      {:decimal, "~> 2.0"},
      {:ecto_sql, "~> 3.0"},     # <= New line
      {:ecto_enum, "~> 1.4"},    # <= New line
      {:phoenix_pubsub, "~> 2.0"},
      {:postgrex, ">= 0.0.0"},   # <= New line
      {:streamer, in_umbrella: true}
    ]
  end
```

Remember about installing those deps using:

```{r, engine = 'bash', eval = FALSE}
$ mix deps.get
```

We can now use the ecto generator to add an the ecto repository to the Naive application:

```{r, eval = FALSE}
$ cd apps/naive
$ mix ecto.gen.repo -r Naive.Repo
* creating lib/naive
* creating lib/naive/repo.ex
* updating ../../config/config.exs
Don't forget to add your new repo to your supervision tree
(typically in lib/naive/application.ex):

    {Naive.Repo, []}

And to add it to the list of Ecto repositories in your
configuration files (so Ecto tasks work as expected):

    config :naive,
      ecto_repos: [Naive.Repo]
```

Back to the IDE, the generator updated our `config/config.exs` file with the default access details to the database, we need to modify them to point to our Postgres docker instance as well as add a list of ecto repositories for our naive app (as per instruction above):

```{r, engine = 'elixir', eval = FALSE}
# /config/config.exs
config :naive,                # <= added line
  ecto_repos: [Naive.Repo],   # <= added line
  binance_client: BinanceMock # <= merged from existing config

config :naive, Naive.Repo,
  database: "naive",                  # <= updated
  username: "postgres",               # <= updated
  password: "hedgehogSecretPassword", # <= updated
  hostname: "localhost"
```

Here we can use `localhost` as inside the `docker-compose.yml` file we defined port forwarding from the container to the host(Postgres is available at localhost:5432). We also merged the existing `binance_client` setting together with the new `ecto_repos` setting.

The last step to be able to communicate with the database using `Ecto` will be to add the `Naive.Repo` module(created by generator) to the children list of the `Naive.Application`:

```{r, engine = 'elixir', eval = FALSE}
# /apps/naive/lib/naive/application.ex
...
  def start(_type, _args) do
    children = [
      {Naive.Repo, []}, # <= added line
      {
        DynamicSupervisor,
        strategy: :one_for_one, name: Naive.DynamicSymbolSupervisor
      }
    ]
    ...
```

## Create and migrate the DB

We can now create a new naive database using the `mix` tool, after that we will be able to generate a migration file that will create the `settings` table:

```{r, engine = 'bash', eval = FALSE}
$ mix ecto.create -r Naive.Repo
The database for Naive.Repo has been created
$ cd apps/naive
$ mix ecto.gen.migration create_settings
* creating priv/repo/migrations
* creating priv/repo/migrations/20210202223209_create_settings.exs
```

We can now copy the current hardcoded settings from the `Naive.Leader` module and use them as a column list of our new `settings` table. All of the below alterations need to be done inside the `change` function of our migration file:

```{r, engine = 'elixir', eval = FALSE}
# /apps/naive/priv/repo/migrations/20210202223209_create_settings.exs
...
  def change do
    create table(:settings) do
      add(:symbol, :text, null: false)
      add(:chunks, :integer, null: false)
      add(:budget, :decimal, null: false)
      add(:buy_down_interval, :decimal, null: false)
      add(:profit_interval, :decimal, null: false)
      add(:rebuy_interval, :decimal, null: false)
    end
  end
```

At this moment we just copied the settings and converted them to columns using the `add` function. We need now to take care of the `id` column. We need to pass `primary_key: false` to the `create table` macro to stop it from creating the default integer-based `id` column. Instead of that we will define the `id` column ourselves with `:uuid` type and pass a flag that will indicate that it's the primary key of the `settings` table:

```{r, engine = 'elixir', eval = FALSE}
# /apps/naive/priv/repo/migrations/20210202223209_create_settings.exs
...
    create table(:settings, primary_key: false) do
      add(:id, :uuid, primary_key: true)
      ...
```

\newpage

We will also add create and update timestamps that come as a bundle when using the `timestamps()` function inside the `create table` macro:

```{r, engine = 'elixir', eval = FALSE}
# /apps/naive/priv/repo/migrations/20210202223209_create_settings.exs
...
    create table(...) do
      ...

      timestamps() # <= both create and update timestamps
    end
    ...
```

We will add a unique index on the symbol column to avoid any possible duplicates:

```{r, engine = 'elixir', eval = FALSE}
# /apps/naive/priv/repo/migrations/20210202223209_create_settings.exs
...
    create table(...) do
      ...
    end

    create(unique_index(:settings, [:symbol]))
  end
...
```

We will now add the `status` field which will be an Enum. It will be defined inside a separate file and `alias`'ed from our migration, this way we will be able to use it from within the migration and the inside the `lib` code. First, we will apply changes to our migration and then we will move on to creating the Enum module.
Here's the full implementation of migration for reference:

```{r, engine = 'elixir', eval = FALSE}
# /apps/naive/priv/repo/migrations/20210202223209_create_settings.exs
defmodule Naive.Repo.Migrations.CreateSettings do
  use Ecto.Migration

  alias Naive.Schema.TradingStatusEnum

  def change do
    TradingStatusEnum.create_type()

    create table(:settings, primary_key: false) do
      add(:id, :uuid, primary_key: true)
      add(:symbol, :text, null: false)
      add(:chunks, :integer, null: false)
      add(:budget, :decimal, null: false)
      add(:buy_down_interval, :decimal, null: false)
      add(:profit_interval, :decimal, null: false)
      add(:rebuy_interval, :decimal, null: false)
      add(:status, TradingStatusEnum.type(), default: "off", null: false)

      timestamps()
    end

    create(unique_index(:settings, [:symbol]))
  end
end
```

That finishes our work on the migration file. We will now focus on `TradingStatusEnum` implementation. First, we need to create a `schema` directory inside the `apps/naive/lib/naive` directory and file called `trading_status_enum.ex` and place below logic (defining the enum) in it:

```{r, engine = 'elixir', eval = FALSE}
# /apps/naive/lib/naive/schema/trading_status_enum.ex
import EctoEnum

defenum(Naive.Schema.TradingStatusEnum, :trading_status, [:on, :off])
```

We used the `defenum` macro from the `ecto_enum` module to define our enum. It's interesting to point out that we didn't need to define a new module as `defenum` macro takes care of that for us.

Let's run the migration to create the table, unique index, and the enum:

```{r, engine = 'bash', eval = FALSE}
$ mix ecto.migrate
00:51:16.757 [info]  == Running 20210202223209 Naive.Repo.Migrations.CreateSettings.change/0
forward
00:51:16.759 [info]  execute "CREATE TYPE public.trading_status AS ENUM ('on', 'off')"
00:51:16.760 [info]  create table settings
00:51:16.820 [info]  create index settings_symbol_index
00:51:16.829 [info]  == Migrated 20210202223209 in 0.0s
```

We can now create a schema file for the `settings` table so inside the `/apps/naive/lib/naive/schema` create a file called `settings.ex`. We will start with a skeleton implementation of schema file together with the copied list of columns from the migration and convert to `ecto`'s types using it's [docs](https://hexdocs.pm/ecto/Ecto.Schema.html#module-primitive-types):

```{r, engine = 'elixir', eval = FALSE}
# /apps/naive/lib/naive/schema/settings.ex
defmodule Naive.Schema.Settings do
  use Ecto.Schema

  alias Naive.Schema.TradingStatusEnum

  @primary_key {:id, :binary_id, autogenerate: true}

  schema "settings" do
    field(:symbol, :string)
    field(:chunks, :integer)
    field(:budget, :decimal)
    field(:buy_down_interval, :decimal)
    field(:profit_interval, :decimal)
    field(:rebuy_interval, :decimal)
    field(:status, TradingStatusEnum)

    timestamps()
  end
end
```

## Seed symbols' settings

So we have all the pieces of implementation to be able to create DB, migrate the `settings` table, and query it using Ecto. To be able to drop the hardcoded settings from the `Naive.Leader` we will need to fill our database with the "default" setting for each symbol. To achieve that we will define default settings inside the `config/config.exs` file and we will create a seed script that will fetch all pairs from Binance and insert a new config row inside DB for each one.

Let's start by adding those default values to the config file(we will merge them into the structure defining `binance_client` and `ecto_repos`):

```{r, engine = 'elixir', eval = FALSE}
# config/config.exs
config :naive,
  ecto_repos: [Naive.Repo],
  binance_client: BinanceMock,
  trading: %{
    defaults: %{
      chunks: 5,
      budget: 1000,
      buy_down_interval: "0.0001",
      profit_interval: "-0.0012",
      rebuy_interval: "0.001"
    }
  }
```

\newpage

Moving on to the seeding script, we need to create a new file called `seed_settings.exs` inside the                           
`/apps/naive/lib/naive/priv/` directory. Let's start by aliasing the required modules and requiring the `Logger`:

```{r, engine = 'elixir', eval = FALSE}
# /apps/naive/priv/seed_settings.exs
require Logger

alias Decimal
alias Naive.Repo
alias Naive.Schema.Settings
```

Next, we will get the Binance client from the config:

```{r, engine = 'elixir', eval = FALSE}
# /apps/naive/priv/seed_settings.exs
...
binance_client = Application.compile_env(:naive, :binance_client)
```



Now, it's time to fetch all the symbols(pairs) that Binance supports:

```{r, engine = 'elixir', eval = FALSE}
# /apps/naive/priv/seed_settings.exs
...
Logger.info("Fetching exchange info from Binance to create trading settings")

{:ok, %{symbols: symbols}} = binance_client.get_exchange_info()
```

Now we need to fetch default trading settings from the config file as well as the current timestamp:

```{r, engine = 'elixir', eval = FALSE}
# /apps/naive/priv/seed_settings.exs
...
%{
  chunks: chunks,
  budget: budget,
  buy_down_interval: buy_down_interval,
  profit_interval: profit_interval,
  rebuy_interval: rebuy_interval
} = Application.compile_env(:naive, :trading).defaults

timestamp = NaiveDateTime.utc_now()
  |> NaiveDateTime.truncate(:second)
```

We will use the default settings for all rows to be able to insert data into the database. Normally we wouldn't need to set `inserted_at` and `updated_at` fields as Ecto would generate those values for us when using `Repo.insert/2` but we won't be able to use this functionality as it takes a *single* record at the time. We will be using `Repo.insert_all/3` which is a bit more low-level function without those nice features like filling timestamps(sadly). Just to be crystal clear - `Repo.insert/2` takes *at least a couple of seconds*(on my machine) for 1000+ symbols currently supported by Binance, on the other hand `Repo.insert_all/3`, will insert all of them in a couple of hundred milliseconds.

As our structs will differ by only the `symbol` column we can first create a full struct that will serve as a template:

```{r, engine = 'elixir', eval = FALSE}
# /apps/naive/priv/seed_settings.exs
...
base_settings = %{
  symbol: "",
  chunks: chunks,
  budget: Decimal.new(budget),
  buy_down_interval: Decimal.new(buy_down_interval),
  profit_interval: Decimal.new(profit_interval),
  rebuy_interval: Decimal.new(rebuy_interval),
  status: "off",
  inserted_at: timestamp,
  updated_at: timestamp
}
```



We will now map each of the retrieved symbols and inject them to the `base_settings` structs and pushing all of those to the `Repo.insert_all/3` function:

```{r, engine = 'elixir', eval = FALSE}
# /apps/naive/priv/seed_settings.exs
...
Logger.info("Inserting default settings for symbols")

maps = symbols
  |> Enum.map(&(%{base_settings | symbol: &1["symbol"]}))

{count, nil} = Repo.insert_all(Settings, maps)

Logger.info("Inserted settings for #{count} symbols")
```

## Update the `Naive.Leader` to fetch settings

The final step will be to update the `Naive.Leader` to fetch the settings from the database. At the top of the module add the following:

```{r, engine = 'elixir', eval = FALSE}
# /apps/naive/lib/naive/leader.ex
  ...
  alias Naive.Repo
  alias Naive.Schema.Settings
  ...
```

Now we need to modify the `fetch_symbol_settings/1` to fetch settings from DB instead of the hardcoded map. We will use `Repo.get_by!/3` as we are unable to trade without settings. The second trick used here is `Map.from_struct/1` that is required here as otherwise result would become the `Naive.Schema.Settings` struct(this would cause problems further down the line as we are iterating on the returned map and would get the `protocol Enumerable not implemented for %Naive.Schema.Settings` error):

```{r, engine = 'elixir', eval = FALSE}
# /apps/naive/lib/naive/leader.ex
  ...
  defp fetch_symbol_settings(symbol) do
    symbol_filters = fetch_symbol_filters(symbol)
    settings = Repo.get_by!(Settings, symbol: symbol)

    Map.merge(
      symbol_filters,
      settings |> Map.from_struct()
    )
  end
  ...
```

We can now run the seeding script to fill our database with the default settings:

```{r, engine = 'bash', eval = FALSE}
$ cd apps/naive
$ mix run priv/seed_settings.exs
18:52:29.341 [info]  Fetching exchange info from Binance to create trading settings
18:52:31.571 [info]  Inserting default settings for symbols
18:52:31.645 [info]  Inserted settings for 1276 symbols
```

We can verify that records were indeed inserted into the database by connecting to it using the `psql` application:

```{r, engine = 'bash', eval = FALSE}
$ psql -Upostgres -hlocalhost
Password for user postgres: # <= use 'postgres' password here
...
postgres=# \c naive
You are now connected to database "naive" as user "postgres".
naive=# \x
Expanded display is on.
naive=# SELECT * FROM settings;
-[ RECORD 1 ]-----+-------------------------------------
id                | 159c8f32-d571-47b2-b9d7-38bb42868043
symbol            | ETHUSDT
chunks            | 5
budget            | 1000
buy_down_interval | 0.0001
profit_interval   | -0.0012
rebuy_interval    | 0.001
status            | off
inserted_at       | 2021-02-02 18:52:31
updated_at        | 2021-02-02 18:52:31

# press arrows to scroll, otherwise press `q`

naive=# SELECT COUNT(*) FROM settings;
-[ RECORD 1 ]
count | 1276

naive=# \q # <= to close the `psql`
```

That confirms that there are 1276 settings inside the database that will allow us to continue trading which we can check by running our app inside the IEx(from the main project's directory):

```{r, engine = 'bash', eval = FALSE}
$ iex -S mix
...
iex(1)> Naive.start_trading("NEOUSDT")
19:20:02.936 [info]  Starting new supervision tree to trade on NEOUSDT
{:ok, #PID<0.378.0>}
19:20:04.584 [info]  Initializing new trader(1612293637000) for NEOUSDT
```

The above log messages confirm that the `Naive.Leader` was able to fetch settings from the database that were later put into the `Naive.Trader`'s state and passed to it.

[Note] Please remember to run the `mix format` to keep things nice and tidy.

The source code for this chapter can be found on [GitHub](https://github.com/Cinderella-Man/hands-on-elixir-and-otp-cryptocurrency-trading-bot-source-code/tree/chapter_10)