diff --git a/README-hidden.md b/README-hidden.md deleted file mode 100644 index b38118d..0000000 --- a/README-hidden.md +++ /dev/null @@ -1,154 +0,0 @@ -[![Build Status](https://travis-ci.org/bearmug/erlang-pure-migrations.svg?branch=master)](https://travis-ci.org/bearmug/erlang-pure-migrations) [![Coverage Status](https://coveralls.io/repos/github/bearmug/erlang-pure-migrations/badge.svg?branch=master)](https://coveralls.io/github/bearmug/erlang-pure-migrations?branch=master) - -- [Erlang ❤ pure Postgres migrations](#erlang---pure-postgres-migrations) - * [Quick start](#quick-start) - * [Versioning model](#versioning-model) - + [Versioning strictness](#versioning-strictness) - + [No-downgrades policy](#no-downgrades-policy) - + [Usage with epgsql TBD](#usage-with-epgsql-tbd) - + [Alternative wrappers TBD](#alternative-wrappers-tbd) - * [Purely functional approach](#purely-functional-approach) - + [Purity tool #1: effects externalization](#purity-tool--1--effects-externalization) - + [Purity tool #2: make effects explicit](#purity-tool--2--make-effects-explicit) - + [Used functional programming abstractions](#used-functional-programming-abstractions) - - [Functions composition](#functions-composition) - - [Functor applications](#functor-applications) - - [Partial function applications](#partial-function-applications) - -# Erlang ❤ pure Postgres migrations -This amazing toolkit has [one and only](https://en.wikipedia.org/wiki/Unix_philosophy) -purpose. It is to migrate Postgres database, using Erlang stack. And as an -extra - do this in "no side-effects" mode aka purely functional. - -Please note - library provides migrations engine capabilities only. You -will need to pass two handlers (or functions) there: - * transaction handler, which apen/commit/rollback transactions scope - * queries handler, to run queries against database -For more details see short in-code [specification](https://github.com/bearmug/oleg-migrations/blob/master/src/oleg_engine.erl#L7). - -## Quick start -Just call `engine:migrate/3`, providing: - * `Path` to migration scripts. Those should be versioned strictly - incrementally from starting from 0 over their names with speparating `_` - symbol. For example: `0_create_database.sql` or `005_UPDATE-IMPORTANT-FILED`. - * `FTx` handler to manage transaction scope. See signature details [here](https://github.com/bearmug/oleg-migrations/blob/make-engine-free-of-side-effects/src/oleg_engine.erl#L9) - and also usage examples into integration tests pack. - * `FQuery` to communicate with database, please see signature details, - since library needs to be answered with number or list of strings - -## Versioning model -### Versioning strictness -As mentioned, versioning model is very opinionated and declares itself -as strictly incremental from 0 and upwards. With given approach, there is -no way for conflicting database updates, applied from different pull-requests -or branches (depends on deployment model). -### No-downgrades policy -As you may see, there is **no downgrade feature available**. Please -consider this while evaluating library for your project. This hasn't been -tooling in order to: - * keep tooling as simple as possible, obviously :) - * delegate upcoming upgrades validation to CI with - unit/integration/acceptance tests chain. Decent test pack and reasonable - CI/CD process (metric-based rollouts, monitored environments, controlled - test coverage regression) making database rollback feature virtually - unused. And without healthy and automated CI/CD cycle there are much - more opportunities to break the system. Database rollback opportunity - could be little to no help. - -### Usage with epgsql TBD -### Alternative wrappers TBD - -## Purely functional approach -Oh, **there is more!** Library implemented in the [way](https://en.wikipedia.org/wiki/Pure_function), -that all side-effects either externalized or deferred explicitly. Goals -are quite common and well-known: - * bring side-effects as close to program edges as possible. And - eventually have referential transparency, enhanced code reasoning, better - bugs reproduceability, etc... - * make unit testing as simple as breeze - * library users empowered to re-run idempotent code safely. Well, if - tx/query handlers are real ones - execution is still idempotent (at - application level) and formally pure. But purity maintained inside - library code only. Some query calls are to be done anyway (like migrations - table creation, if this one does not exist). - -### Purity tool #1: effects externalization -There are 2 externalized kind of effects: - * transaction management handler - * database queries handler -Although, those two can`t be pure in real application, it is failrly -simple to replace them with their pure versions if we would like to -(for debug purposes, or testing, or something else). - -### Purity tool #2: make effects explicit -Other effects (file operations, like directory listing or file content -read) are deferred in bulk. This way 2 goals achieved: - * pure actions sequence built and validated without any impact from - external world - * library users decides if regarding moment, when they ready to apply - changes. Maybe for some reason they would like to prepare execution -> - change migrations folder content -> run migrations. - -### Used functional programming abstractions -Sure, Erlang is deeply funcitonal language. But at the same time, for -obvious reasons ( 1)not much people need tools like these 2)it is deadly - simple to implement required abstractions on your own), there are no -(at least I did not manage to find) widely used functional primitives -Erlang library. - -#### Functions composition -Abstraction quite useful if someone would like to compose two functions -without their actual nested execution (or without their application, -alternatively speaking). This pretty standard routine may look like below -(Scala or Kotlin+Arrow): -```scala -val divideByTwo = (number : Int) => number / 2; -val addThree = (number: Int) => number + 3; -val composed = addThree compose divideByTwo -``` -To keep things close to the ground and avoiding infix notation, in -Erlang it is could be represented like: -```erlang -compose(F1, F2) -> - fun() -> F2(F1()) end. -``` -You may find library funcitonal composition example in a few locations -[here](https://github.com/bearmug/oleg-migrations/blob/make-engine-free-of-side-effects/src/oleg_engine.erl#L36). - -#### Functor applications -There area few places in library with clear need to compose function **A** -and another function **B** inside deferred execution context. Specifics is -that **A** supplies list of objects, and **B** should be applied to each of -them. Sounds like some functor **B** to be applied to **A** output, when -this output is being wrapped into future execution context. Two cases -of this appeared in library: - * have functor running and produce nested list of contexts: -```erlang -%% Map/1 call here produces new context (defferred function call in Erlang) -map(Generate, Map) -> - fun() -> [Map(R) || R <- Generate()] end. -``` - * flatten (or fold) contexts to a single one: -```erlang -%% Flatten/1 call compactifies contexts and folds 2 levels to single one -flatten(Generate, Flatten) -> - fun() -> [Flatten(R) || R <- Generate()] end. -``` -#### Partial function applications -This technique is very useful, in case if not all function arguments -known yet. Or maybe there is deliberate decision to pass some of arguments -later on. Again, in Scala it may look like: -```scala -val add = (a: Int, b: Int) => a + b -val partiallyApplied = add(3, _) -``` -Library code has very simplistic partial application, done for exact -arguments number (although it is easy to generalize it for arguments, -represented as list): -```erlang -partial(F, A, B) -> - fun(C) -> F(A, B, C) end. -``` -Exactly this feature helps [here](https://github.com/bearmug/oleg-migrations/blob/make-engine-free-of-side-effects/src/oleg_engine.erl#L19) -to pass particular migration to partially applied function. Therefore, -no need to care about already known parameters. \ No newline at end of file diff --git a/README.md b/README.md new file mode 100644 index 0000000..3d3da38 --- /dev/null +++ b/README.md @@ -0,0 +1,156 @@ +# Erlang ❤ pure database migrations +> Database migrations engine. Effects-free. + +[![Build Status](https://travis-ci.org/bearmug/erlang-pure-migrations.svg?branch=master)](https://travis-ci.org/bearmug/erlang-pure-migrations) [![Coverage Status](https://coveralls.io/repos/github/bearmug/erlang-pure-migrations/badge.svg?branch=master)](https://coveralls.io/github/bearmug/erlang-pure-migrations?branch=master) + +Migrate your Erlang application database with no effort. +This amazing toolkit has [one and only](https://en.wikipedia.org/wiki/Unix_philosophy) +purpose - consistently upgrade database schema, using Erlang stack. +As an extra - do this in "no side-effects" mode. + +# Table of content +- [Current limitations](#current-limitations) +- [Quick start](#quick-start) + * [Compatibility table](#compatibility-table) + * [Live code samples](#live-code-samples) + + [Postgres and epgsql/epgsql sample](#postgres-and--epgsql-epgsql--https---githubcom-epgsql-epgsql--sample) +- [No-effects approach and tools used to achieve it](#no-effects-approach-and-tools-used-to-achieve-it) + * [Tool #1: effects externalization](#tool--1--effects-externalization) + * [Tool #2: make effects explicit](#tool--2--make-effects-explicit) +- [Functional programming abstractions used](#functional-programming-abstractions-used) + * [Functions composition](#functions-composition) + * [Functor applications](#functor-applications) + * [Partial function applications](#partial-function-applications) + +# Current limitations + * **up** transactional migration available only. No **downgrade** + or **rollback** possible. Either whole **up** migration completes OK + or failed and rolled back to the state before migration. + * migrations engine **deliberately isolated from any specific + database library**. This way engine user is free to choose from variety + of frameworks (see tested combinations [here](#compatibility-table)) and so on. + +# Quick start +Just call `engine:migrate/3` (see specification [here](src/engine.erl#L9)), providing: + * `Path` to migration scripts folder (strictly and incrementally enumerated). + * `FTx` transaction handler + * `FQuery` database queries execution handler + +## Compatibility table +| Database dialect | Library | Example | +| -------------- | ------ | ------- | +| postgres | [epgsql/epgsql:4.2.1](https://github.com/epgsql/epgsql/releases/tag/4.2.1) | [epgsql test](test/epgsql_migrations_SUITE.erl) + +## Live code samples +### Postgres and [epgsql/epgsql](https://github.com/epgsql/epgsql) sample +
+ Click to expand + + ```erlang + Conn = ?config(conn, Opts), + MigrationCall = + engine:migrate( + "scripts/folder/path", + fun(F) -> epgsql:with_transaction(Conn, fun(_) -> F() end) end, + fun(Q) -> + case epgsql:squery(Conn, Q) of + {ok, [ + {column, <<"version">>, _, _, _, _, _}, + {column, <<"filename">>, _, _, _, _, _}], Data} -> + [{list_to_integer(binary_to_list(BinV)), binary_to_list(BinF)} || {BinV, BinF} <- Data]; + {ok, [{column, <<"max">>, _, _, _, _, _}], [{null}]} -> -1; + {ok, [{column, <<"max">>, _, _, _, _, _}], [{N}]} -> + list_to_integer(binary_to_list(N)); + [{ok, _, _}, {ok, _}] -> ok; + {ok, _, _} -> ok; + {ok, _} -> ok; + Default -> Default + end + end), + ... + %% more preparation steps + ... + %% migration call + ok = MigrationCall(), + + ``` +Also see examples from live epgsql integration tests +[here](test/epgsql_migrations_SUITE.erl) +
+ +# No-effects approach and tools used to achieve it +Oh, **there is more!** Library implemented in the [way](https://en.wikipedia.org/wiki/Pure_function), +that all side-effects either externalized or deferred explicitly. Reasons +are quite common: + * bring side-effects as close to program edges as possible. Which may + mean enhanced code reasoning, better bugs reproduceability, etc... + * simplify module contracts testing + * library users empowered to re-run idempotent code safely. Well, if + tx/query handlers are real ones - execution is still idempotent (at + application level) and formally pure. But purity maintained inside + library code only. One call is to be issued anyway - migrations + table creation, if this one does not exists. + +## Tool #1: effects externalization +There are 2 externalized kind of effects: + * transaction management handler + * database queries handler +Although, those two can`t be pure in real application, it is fairly +simple to replace them with their pure versions if we would like to +(for debug purposes, or testing, or something else). + +## Tool #2: make effects explicit +Other effects (like file operations) are deferred in bulk with outcome +like: + * pure referentially-transparent program actions composed only. Impact + or any communication with external world postponed until later stages + * library users decide when they ready to apply migration changes. + Maybe for some reason they would like to prepare execution -> + prepare migrations folder content -> run migrations. + +# Functional programming abstractions used +## Functions composition +This trick is quite useful if someone would like to compose two functions +without their actual execution (or without their application, +alternatively speaking). This pretty standard routine may look like below +(Scala or Kotlin+Arrow): +```scala +val divideByTwo = (number : Int) => number / 2; +val addThree = (number: Int) => number + 3; +val composed = addThree compose divideByTwo +``` +Simplistic Erlang version: +```erlang +compose(F1, F2) -> fun() -> F2(F1()) end. +``` + +## Functor applications +There area few places in library with clear need to compose function **A** +and another function **B** inside deferred execution context. Specifics is +that **A** supplies list of objects, and **B** should be applied to each of +them. Sounds like some functor **B** to be applied to **A** output, when +this output is being wrapped into future execution context. Two cases +of this appeared in library: + * have functor running and produce nested list of contexts: +```erlang +%% Map/1 call here produces new context (defferred function call) +map(Generate, Map) -> fun() -> [Map(R) || R <- Generate()] end. +``` + * flatten (or fold) contexts (or function calls) list to a single one: +```erlang +flatten(Generate) -> fun() -> [ok = R() || R <- Generate()], ok end. +``` +## Partial function applications +Partial application is very useful, in case if not all function arguments +known yet. Or maybe there is deliberate decision to pass some of arguments +later on. Again, in Scala it may look like: +```scala +val add = (a: Int, b: Int) => a + b +val partiallyApplied = add(3, _) +``` +Library code has very simplistic partial application, done for exact +arguments number (although it is easy to generalize it for arguments, +represented as list): +```erlang +Partial = fun(V_F) -> do_migration(Path, FQuery, V_F) end, +``` \ No newline at end of file diff --git a/src/dialect_postgres.erl b/src/dialect_postgres.erl index c6bf199..15bf272 100644 --- a/src/dialect_postgres.erl +++ b/src/dialect_postgres.erl @@ -1,6 +1,6 @@ -module(dialect_postgres). --export([init/0, migrations_names/0, save_migration/2, latest_existing_version/0]). +-export([init/0, migrations_done/0, save_migration/2, latest_existing_version/0]). init() -> "CREATE TABLE IF NOT EXISTS database_migrations_history ( @@ -9,7 +9,7 @@ init() -> creation_timestamp TIMESTAMP NOT NULL DEFAULT NOW() )". -migrations_names() -> +migrations_done() -> "SELECT version, filename FROM database_migrations_history". save_migration(Version, Filename) -> diff --git a/src/engine.erl b/src/engine.erl index de52e44..23ceafe 100644 --- a/src/engine.erl +++ b/src/engine.erl @@ -2,13 +2,15 @@ -export([migrate/3]). +-type version() :: non_neg_integer(). +-type migration() :: {version(), Filename :: nonempty_string()}. -type error() :: {error, Type :: atom(), Details :: any()}. -spec migrate( Path :: nonempty_string(), FTx :: fun((F) -> ok | error()), - FQuery :: fun((nonempty_string()) -> ok | list(string()) | integer() | error())) -> R - when + FQuery :: fun((nonempty_string()) -> ok | [migration()] | version() | error())) -> R + when F :: fun(() -> ok | error()), R :: fun(() -> ok | error()). migrate(Path, FTx, FQuery) -> @@ -43,7 +45,7 @@ find_migrations(ScriptsLocation, FQuery) -> fun(Files) -> filter_script_files(Files, ScriptsLocation, FQuery) end). filter_script_files({ok, Files}, _Folder, FQuery) -> - MigrationsDone = sets:from_list(FQuery(dialect_postgres:migrations_names())), + MigrationsDone = sets:from_list(FQuery(dialect_postgres:migrations_done())), lists:filter( fun(N) -> not sets:is_element(N, MigrationsDone) end, lists:keysort(1, [version_and_filename(F) || F <- Files])); diff --git a/test/config/test.config b/test/config/test.config index 273fdde..04811d9 100644 --- a/test/config/test.config +++ b/test/config/test.config @@ -1,11 +1,11 @@ -[{db, [{ - config, [ - {host, "localhost"}, - {port, 5432}, - {database, "migration"}, - {username, "migration"}, - {secret, "migration"}, - {timeout, 10000} - ] - }] +[{postgres, [{ + config, [ + {host, "localhost"}, + {port, 5432}, + {database, "migration"}, + {username, "migration"}, + {secret, "migration"}, + {timeout, 10000} + ] + }] }]. diff --git a/test/epgsql_migrations_SUITE.erl b/test/epgsql_migrations_SUITE.erl index a19df92..40107a0 100644 --- a/test/epgsql_migrations_SUITE.erl +++ b/test/epgsql_migrations_SUITE.erl @@ -150,7 +150,6 @@ transactional_migration_test(Opts) -> epgsql_query_fun(Conn) -> fun(Q) -> case epgsql:squery(Conn, Q) of - ok -> ok; {ok, [ {column, <<"version">>, _, _, _, _, _}, {column, <<"filename">>, _, _, _, _, _}], Data} -> @@ -171,7 +170,7 @@ init_per_testcase(_TestCase, Opts) -> {database, Database}, {username, Username}, {secret, Secret}, - {timeout, Timeout}]} = application:get_env(db, config), + {timeout, Timeout}]} = application:get_env(postgres, config), {ok, C} = epgsql:connect(Host, Username, Secret, #{ database => Database,