Fix driver architecture (#188)

* add some config information, document the optimized_driver option * implement passthrough adapter for specific drivers * implement simple ets driver * move type definitions under record definitions, add type * remove duplicate type * add data_model option and documentation * remove unused handlers * rename header files * add new options and defaults * add config/fmke.config * refactor module to work with an adapter passed in by a supervisor * update header file name * update header file name * change default adapter * add test cases for ets with nested and non_nested model, refactor code for comparing records * implement simple ets driver * create adapter behaviour * implement key-value driver behaviour * implement sup responsible for adapter, driver, conn pool * add adapter for key value stores * add sup for managing several connection pools * use supervision tree instead of single supervisor * implement simple driver for antidote * delete modules from old architecture * add tools lib to enable runtime profiling and monitoring * fix indentation * update type definitions for get functions * update type definitions * add function specs, change encode_string_list, add test * update prescription id types * fix wrong return type * make return match behaviour, match only return types from fetch types * add runtime_tools for profiling * fix order of arguments passed into gen_server:start_link * add json test for list of prescription objects * make return match behaviour * update return values on successful commit * update value for successful commit, fix wrong start return value * rename antidote optimized driver * rename riak optimized driver * improve fmke_test_utils API * add more compare presc variants, refactor group listing * use updated test utils API * remove old riak driver * make driver use gen_server behaviour, dialyzer is happy * use only driver part of init args * rename test utils to fmke_test_setup * implement comparison functions for FMKe entities in a separate module * use comparison functions from fmke_test_utils, refactor * continue refactor to use utils entity comparison * allow for connection manager not to be up, as this is allowed in certain deployments * add clause for json prescription object * add decode function, useful for testing * address case where one of the prescription objects is in json format * adapt to new driver architecture, use test_utils library for comparison * fix parse_csv_string, parse_string * add test for decode prescription * tweak elvis rules * add type definitions for records, as suggested by elvis * use entity types instead of records * use entity types instead of records, fix bug in test case * use header file and entity types in callback definitions * use entity types, fix corner case in comparison, add tests for entity comparison * fix bugs in prescription comparison * update README * fix #187 * remove duplicate type * update opt driver * remove bench dependencies * comment module contents, will be updated soon * remove multiple releases tests, should be tested by client * reduce number of processes * update travis script * enable dialyzer checks * implement update prescription operations with object history * add tests to cover get_[processed_]prescriptions cases * fix several bugs found in tests, add context to write operations * indent code * fix bugs found in tests * rename module * add spec to gen_pool_name/2 * fix #187 * fix #189 * fix function spec * fix #190 * implement redis driver, still need to fix dialyzer and linter warnings * fix dialyzer and lint issues * remove debug output
goncalotomas · May 12, 2018 · f8a7a83 · f8a7a83
1 parent e1e6331
commit f8a7a83
Show file tree

Hide file tree

Showing 50 changed files with 5,079 additions and 3,825 deletions.
diff --git a/.gitignore b/.gitignore
@@ -19,3 +19,4 @@ rebar3.crashdump
 tests/
 .DS_Store
 .idea/
+config/fmke.config
diff --git a/.travis.yml b/.travis.yml
@@ -9,12 +9,12 @@ install:
   - make all
   - make lint
   - make xref
+  - make dialyzer
 before_script:
   - epmd -daemon
   - sudo sh -c 'echo 0 > /proc/sys/net/ipv6/conf/all/disable_ipv6'
 script:
   - make test
-  - make test-multiple-releases
   - rebar3 as test coveralls send
   # - make bench
 after_failure:

diff --git a/Makefile b/Makefile
@@ -1,4 +1,4 @@
-REBAR = rebar3
+REBAR=rebar3
 BENCH=_build/test/lib/lasp_bench
 
 all: compile rel
@@ -33,7 +33,6 @@ bench-riak-norm: rel
 
 compile:
 	${REBAR} as test compile
-	cd ./_build/test/lib/lasp_bench && rebar3 escriptize
 
 console: rel
 	./_build/default/rel/fmke/bin/env console
@@ -87,7 +86,7 @@ shell-riak: rel
 	./scripts/start_data_store.sh riak
 	./_build/default/rel/fmke/bin/env console
 
-start:
+start: rel
 	./scripts/start_fmke.sh
 
 start-antidote: select-antidote
@@ -121,25 +120,5 @@ test: all
 	rebar3 eunit
 	rebar3 ct
 
-test-multiple-releases:
-	rm -rf _build/default/rel
-	./scripts/start_data_store.sh antidote
-	./scripts/config/change_http_port.sh 9090
-	./scripts/config/change_db.sh antidote
-	./scripts/config/change_db_ports.sh antidote
-	${REBAR} release -n fmke
-	./_build/default/rel/fmke/bin/env start
-	sleep 10
-	./scripts/config/change_http_port.sh 9190
-	${REBAR} release -n fmke_test
-	./_build/default/rel/fmke_test/bin/env_test start
-	sleep 10
-	./scripts/populate_fmke.escript "antidote" "../config/benchmark_short.config" "fmke_test@127.0.0.1"
-	_build/test/lib/lasp_bench/_build/default/bin/lasp_bench config/benchmark_short.config
-	./scripts/config/change_http_port.sh 9090
-	./_build/default/rel/fmke/bin/env stop
-	./_build/default/rel/fmke_test/bin/env_test stop
-	./scripts/stop_data_store.sh antidote
-
 xref:
 	rebar3 xref
diff --git a/README.md b/README.md
@@ -1,4 +1,8 @@
-# FMKe [![Build Status](https://travis-ci.org/goncalotomas/FMKe.svg?branch=master)](https://travis-ci.org/goncalotomas/FMKe) [![Coverage Status](https://coveralls.io/repos/github/goncalotomas/FMKe/badge.svg?branch=master)](https://coveralls.io/github/goncalotomas/FMKe?branch=master)
+# FMKe
+![Erlang Version](https://img.shields.io/badge/Erlang%2FOTP-20-brightgreen.svg)
+[![Build Status](https://travis-ci.org/goncalotomas/FMKe.svg?branch=master)](https://travis-ci.org/goncalotomas/FMKe)
+[![Coverage Status](https://coveralls.io/repos/github/goncalotomas/FMKe/badge.svg?branch=master)](https://coveralls.io/github/goncalotomas/FMKe?branch=master)
+![Dialyzer Enabled](https://img.shields.io/badge/dialyzer-enabled-brightgreen.svg)  
 
 FMKe is an extendable real world benchmark for distributed key-value stores.  
 This repository contains code for the application server and a set of scripts for orchestrating deployment and local execution of micro-benchmarks.
@@ -33,13 +37,10 @@ Firstly, separating the application server from the workload generation componen
 We have a generic interface for key-value stores (implemented as an Erlang behaviour) that is well specified, which makes supporting a new database as simple as writing a driver for it. Furthermore, pull requests with new drivers or optimizations for existing ones are accepted and welcomed.
 
 ## Supported data stores
-- AntidoteDB (using nested CRDTs)
-- AntidoteDB (with a normalized data model)
-- Riak (using nested CRDTs)
-- Redis (with a normalized data model)
-### In a future release
-- Lasp
-
+- AntidoteDB
+- Cassandra
+- Redis
+- Riak KV
 
 ## How the benchmark is deployed
 By default FMKe keeps a connection pool to a single database node, and the workload generation is performed by [Lasp Bench][4].  

diff --git a/config/fmke.config b/config/fmke.config
@@ -3,22 +3,46 @@
 %% (1): ["192.168.1.1", "192.168.1.2"]
 %% (2): [{192,168,1,1} {192,168,1,2}]
 %% (3): "192.168.1.1 192.168.1.2"
+%% If you're connecting to multiple nodes on the same port, you can list all of the different addresses here and leave
+%% a single entry in the database_ports option (e.g. [8087]).
 {database_addresses, ["127.0.0.1"]}.
 
 %% List of ports where FMKe should try to connect at boot time.
 %% NOTE: Must use one of the following structures:
 %% (1): ["8080", "8081"]
 %% (2): [8080, 8081]
 %% (3): "8080 8081"
+%% If you're connecting to multiple nodes on the same address but on different ports, you can list all of the different
+%% ports here and leave a single entry in the database_addresses option (e.g. ["127.0.0.1"]).
 {database_ports, [8087]}.
 
 %% Target back end data store. This is required in order for FMKe to load the
 %% correct drivers to connect to your desired data store.
 %% Currently FMKe supports the following data stores:
-%% antidote, riak_kv
+%% antidote, riak, redis
 %% Please select one of the previous values in the form of an erlang atom
 %% (e.g. riak) or string (e.g. "riak")
-{target_database, antidote}.
+{target_database, riak}.
+
+%% Uses an optimized driver that implements the entire FMKe API (only if implemented).
+%% Currently these are available for antidote, riak and redis. You're welcome to implement a new one and submit a
+%% pull request!
+%% Predictably, this defaults to false, so you only need to specify this option when you want to use an optimized driver
+%% implementation.
+% {optimized_driver, true}.
+
+%% Changes the data model, if the available drivers support it.
+%% For key value stores, we consider the possible values to be nested or non_nested, depending on whether the database
+%% keeps references to other objects, or copies of other objects that must be updated on each update to the original.
+%% The gen_fmke_kv_driver behaviour specifies the start/1 callback for all drivers implemented with this behaviour, and
+%% the argument to the start function is the value of this option. We recommend you also implement the gen_server
+%% behaviour and keep it in the state, or put it inside an ETS table (but you must check it's value for each relevant
+%% get or put operation).
+%% When implementing drivers it might make sense to look at how other previous ones were made and attempt to replicate
+%% the logic inside them.
+%% NOTE: Usually optimized drivers only support one data model (the one that yields the best performance), so please
+%% keep this line commented when measuring performance with optimized drivers.
+% {data_model, nested}.
 
 %% When FMKe connects to the database you choose it opens a pool of connections.
 %% This parameter configures the connection pool size.
@@ -28,6 +52,6 @@
 %% open 15 connections to each database node.
 {connection_pool_size, 30}.
 
-%% The HTTP port to which the FMKe HTTP server binds to. Running on a system
-%% reserved port interval (1-2048) will require superuser privileges.
+%% The port on which the FMKe HTTP server binds to. Running on a system
+%% reserved port (0-1023) will require superuser privileges.
 {http_port, 9090}.
diff --git a/elvis.config b/elvis.config
@@ -12,13 +12,13 @@
                   {elvis_style, macro_names, #{ignore => []}},
                   {elvis_style, macro_module_names},
                   {elvis_style, operator_spaces, #{rules => [
-                    {right, ","}, {right, "++"}, {left, "++"}, {right, "--"}, {left, "--"}
+                    {right, "++"}, {left, "++"}, {right, "--"}, {left, "--"}
                   ]}},
-                  {elvis_style, nesting_level, #{level => 3}},
+                  {elvis_style, nesting_level, #{level => 4}},
                   {elvis_style, god_modules, #{limit => 30, ignore => []}},
                   {elvis_style, no_if_expression},
                   {elvis_style, invalid_dynamic_call, #{ignore => [
-                    fmke, fmke_kv_driver, fmke_db_connection
+                    fmke, fmke_db_connection, fmke_pt_adapter, fmke_kv_adapter
                   ]}},
                   {elvis_style, used_ignored_variable},
                   {elvis_style, no_behavior_info},
@@ -27,8 +27,8 @@
                   % {elvis_style, state_record_and_type, disable},
                   {elvis_style, state_record_and_type},
                   {elvis_style, no_spec_with_records},
-                  {elvis_style, dont_repeat_yourself, #{min_complexity => 30}},
-                  {elvis_style, no_debug_call, #{ignore => [fmke_test_utils]}}],
+                  {elvis_style, dont_repeat_yourself, #{min_complexity => 40}},
+                  {elvis_style, no_debug_call, #{ignore => [fmke_test_setup]}}],
                   %% sequential reporter calls other reporters dynamically
                   % {elvis_style, invalid_dynamic_call, #{ignore => [oc_sequential_reporter]}}],
 
@@ -39,10 +39,13 @@
         rules => [{elvis_style, god_modules, #{ignore => [egithub]}},
                   {elvis_style, dont_repeat_yourself, #{min_complexity => 30}},
                   {elvis_style, line_length, #{limit => 120}},
+                  {elvis_style, operator_spaces, #{rules => [
+                    {right, "++"}, {left, "++"}, {right, "--"}, {left, "--"}
+                  ]}},
                   %% be a bit more lax on naming for tests
                   {elvis_style, variable_naming_convention, #{regex => "^_{0,2}([A-Z][0-9a-zA-Z]*)$"}},
                   {elvis_style, state_record_and_type, disable},
-                  {elvis_style, no_debug_call, #{ignore => [fmke_test_utils]}}],
+                  {elvis_style, no_debug_call, #{ignore => [fmke_test_setup]}}],
         ruleset => erl_files
       },
       #{dirs => ["."],

diff --git a/include/fmke.hrl b/include/fmke.hrl
@@ -1,43 +1,44 @@
 -define (APP, fmke).
--define (OPTIONS, [adapter, connection_pool_size, database_addresses, database_ports, http_port, target_database]).
+-define (OPTIONS, [
+    adapter, connection_pool_size, database_addresses, database_ports, http_port,
+    target_database, data_model, optimized_driver
+]).
 -define (DEFAULTS, #{
-    adapter => fmke_ndm_adapter,
+    adapter => fmke_kv_adapter,
     connection_pool_size => 64,
     database_addresses => ["127.0.0.1"],
     database_ports => [8087],
     http_port => 9090,
-    target_database => antidote
+    target_database => riak,
+    data_model => nested,
+    optimized_driver => false
 }).
 
--define (TIMEOUT, 60000).
+-define(TIMEOUT, 60000).
 
--define (CONFIG_FILE_PATH, "/config/fmke.config").
+-define(CONFIG_FILE_PATH, "/config/fmke.config").
+-define(ETS_TABLE_NAME, fmke_ets).
 
 %% TODO move this to an ETS table
 -define(SUPPORTED_DBS, [antidote, antidote_norm, riak_kv, riak_kv_norm, redis]).
 -define(SUPPORTED_KVS, [antidote, antidote_norm, riak_kv, riak_kv_norm, redis]).
 
--type id() :: non_neg_integer().
--type field() :: binary().
--type reason() :: term().
--type crdt() :: term().
-
 -record(prescription, {
-    id :: id()
-    ,patient_id :: id()
-    ,pharmacy_id :: id()
-    ,prescriber_id :: id()
+    id :: id() | binary()
+    ,patient_id :: id() | binary()
+    ,pharmacy_id :: id() | binary()
+    ,prescriber_id :: id() | binary()
     ,date_prescribed :: field()
     ,date_processed = <<"undefined">> :: field()
-    ,drugs :: list(field())
+    ,drugs :: list(string() | binary())
     ,is_processed = <<"prescription_not_processed">> :: field()
 }).
 
 -record(patient, {
     id :: id()
     ,name :: string()
     ,address :: string()
-    ,prescriptions = [] :: list(#prescription{})
+    ,prescriptions = [] :: list(#prescription{} | key())
     % ,treatments=[] :: list(#treatment{})
     % ,events=[] :: list(#event{})
 }).
@@ -46,7 +47,7 @@
     id :: id()
     ,name :: string()
     ,address :: string()
-    ,prescriptions = [] :: list(#prescription{})
+    ,prescriptions = [] :: list(#prescription{} | key())
 }).
 
 -record(facility, {
@@ -63,5 +64,23 @@
     ,name :: string()
     ,address :: string()
     ,speciality :: string()
-    ,prescriptions = [] :: list(#prescription{})
+    ,prescriptions = [] :: list(#prescription{} | key())
 }).
+
+-type id() :: non_neg_integer().
+-type field() :: binary().
+-type reason() :: term().
+-type crdt() :: term().
+-type app_record() :: #facility{} |
+                      #patient{} |
+                      #pharmacy{} |
+                      #prescription{} |
+                      #staff{}.
+-type entity() :: facility | patient | pharmacy | prescription | staff.
+-type key() :: binary().
+
+-type facility() :: #facility{}.
+-type patient() :: #patient{}.
+-type pharmacy() :: #pharmacy{}.
+-type prescription() :: #prescription{}.
+-type staff() :: #staff{}.
diff --git a/include/fmk_http.hrl → include/fmke_http.hrl b/include/fmk_http.hrl → include/fmke_http.hrl
diff --git a/include/fmk_kv.hrl → include/fmke_kv.hrl b/include/fmk_kv.hrl → include/fmke_kv.hrl
diff --git a/rebar.config b/rebar.config
@@ -1,6 +1,6 @@
 {deps, [
     %% Hex dependencies
-    {eredis, "~>1.1"},
+    {eredis_cluster, "~>0.5"},
     {jsx, "~>2.8"},
     {lager, "~>3.5"},
     {poolboy, "~>1.5"},
@@ -34,13 +34,6 @@
         {erl_opts, [no_debug_info, warnings_as_errors]},
         {relx, [{dev_mode, false}]}
     ]},
-    {test, [
-        {erl_opts, [debug_info]},
-        {deps, [
-            {hackney, "1.9.0"},
-            {lasp_bench, {git, "https://github.com/lasp-lang/lasp-bench", {tag, "fmke"}}}
-        ]}
-    ]},
     {lint,  [
         {plugins, [
             {rebar3_lint, {git, "https://github.com/project-fifo/rebar3_lint.git", {tag, "v0.1.9"}}}
@@ -57,18 +50,7 @@
                {copy, "config/fmke.config", "config/fmke.config"}
            ]},
            {overlay_vars, "config/vars.config"},
-           {extended_start_script, true}]},
-
-         {release, {fmke_test, "0.1.0"}, [fmke], [
-            {vm_args, "config/vm_test.args"},
-            {dev_mode, true},
-            {include_erts, false},
-            {overlay, [
-                {copy, "bin/env_test", "bin"},
-                {copy, "config/fmke.config", "config/fmke.config"}
-            ]},
-            {overlay_vars, "config/vars_test.config"},
-            {extended_start_script, true}]}
+           {extended_start_script, true}]}
        ]}.
 
 {xref_checks, [

diff --git a/rebar.lock b/rebar.lock
@@ -3,12 +3,13 @@
   {git,"https://github.com/SyncFree/antidote_pb",
        {ref,"09a2f884a6e0b388475576701b096741d48f48fb"}},
   0},
- {<<"cowboy">>,{pkg,<<"cowboy">>,<<"2.2.0">>},0},
- {<<"cowlib">>,{pkg,<<"cowlib">>,<<"2.1.0">>},1},
- {<<"eredis">>,{pkg,<<"eredis">>,<<"1.1.0">>},0},
+ {<<"cowboy">>,{pkg,<<"cowboy">>,<<"2.4.0">>},0},
+ {<<"cowlib">>,{pkg,<<"cowlib">>,<<"2.3.0">>},1},
+ {<<"eredis">>,{pkg,<<"eredis">>,<<"1.1.0">>},1},
+ {<<"eredis_cluster">>,{pkg,<<"eredis_cluster">>,<<"0.5.11">>},0},
  {<<"goldrush">>,{pkg,<<"goldrush">>,<<"0.1.9">>},1},
  {<<"jsx">>,{pkg,<<"jsx">>,<<"2.9.0">>},0},
- {<<"lager">>,{pkg,<<"lager">>,<<"3.6.0">>},0},
+ {<<"lager">>,{pkg,<<"lager">>,<<"3.6.2">>},0},
  {<<"meck">>,
   {git,"https://github.com/basho/meck.git",
        {ref,"dde759050eff19a1a80fd854d7375174b191665d"}},
@@ -18,21 +19,22 @@
   {git,"git://github.com/basho/erlang_protobuffs.git",
        {ref,"0dde9d3b37b7bec3a4dfb0e87684dd7039e169ab"}},
   2},
- {<<"ranch">>,{pkg,<<"ranch">>,<<"1.4.0">>},1},
+ {<<"ranch">>,{pkg,<<"ranch">>,<<"1.5.0">>},1},
  {<<"riak_client">>,{pkg,<<"riak_client">>,<<"2.5.3">>},0},
  {<<"riak_pb">>,
   {git,"git://github.com/syncfree/riak_pb",
        {ref,"322309b2bb69b38191a51b21247a71b5e412e656"}},
   1}]}.
 [
 {pkg_hash,[
- {<<"cowboy">>, <<"5233ED78AFE96B19EFD33B7BBCAFD15E064B187C0C131C61D50FE69D88DC76FD">>},
- {<<"cowlib">>, <<"F73658B93DD043AF40400C3E4FD997068EBD0C617F8C8F4CD003A1A78EBF94F5">>},
+ {<<"cowboy">>, <<"F1B72FABE9C8A5FC64AC5AC85FB65474D64733D1DF52A26FAD5D4BA3D9F70A9F">>},
+ {<<"cowlib">>, <<"BBD58EF537904E4F7C1DD62E6AA8BC831C8183CE4EFA9BD1150164FE15BE4CAA">>},
  {<<"eredis">>, <<"8D8D74496F35216679B97726B75FB1C8715E99DD7F3EF9F9824A2264C3E0AAC0">>},
+ {<<"eredis_cluster">>, <<"383E44C47B4DF06145448680D84BBC95A7E8CD708E9100BEB0E461ECFAB506FE">>},
  {<<"goldrush">>, <<"F06E5D5F1277DA5C413E84D5A2924174182FB108DABB39D5EC548B27424CD106">>},
  {<<"jsx">>, <<"D2F6E5F069C00266CAD52FB15D87C428579EA4D7D73A33669E12679E203329DD">>},
- {<<"lager">>, <<"0C5BDB1E5937ECA17D7CAB15DD52AA30B9EB12D48A2D9E774EF6385E48709A98">>},
+ {<<"lager">>, <<"2C72950A0E75FFD927DD35BC85EA30E7680A264C4BBD08B328250871AC456CC2">>},
  {<<"poolboy">>, <<"6B46163901CFD0A1B43D692657ED9D7E599853B3B21B95AE5AE0A777CF9B6CA8">>},
- {<<"ranch">>, <<"10272F95DA79340FA7E8774BA7930B901713D272905D0012B06CA6D994F8826B">>},
+ {<<"ranch">>, <<"F04166F456790FEE2AC1AA05A02745CC75783C2BFB26D39FAF6AEFC9A3D3A58A">>},
  {<<"riak_client">>, <<"6B78373E576D80C7020BCBDA3B2C102221D32863580404F59D8791E5CDC19E79">>}]}
 ].