Amount of money on a bank account becomes negative in a Jepsen test

[ligurio]

Bug description

A clear and concise description of what the bug is.

• OS: Linux
• OS Version: Ubuntu 18.04 x86_64
• Architecture amd64

Tarantool 2.8.1-20-g6e1d4a463
Target: Linux-x86_64-RelWithDebInfo
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr -DENABLE_BACKTRACE=ON
Compiler: /usr/bin/cc /usr/lib/ccache/g++
C_FLAGS:-g -O2 -fdebug-prefix-map=/build/tarantool-2.8.1.20=. -specs=/usr/share/dpkg/no-pie-compile.specs -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -std=c11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-gnu-alignof-expression -fno-gnu89-inline -Wno-cast-function-type
CXX_FLAGS:-g -O2 -fdebug-prefix-map=/build/tarantool-2.8.1.20=. -specs=/usr/share/dpkg/no-pie-compile.specs -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -std=c++11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-invalid-offsetof -Wno-gnu-alignof-expression -Wno-cast-function-type

Steps to reproduce

lein run test --workload bank-multitable-lua --nodes-file nodes --username ubuntu --version 2.8 --leave-db-running --time-limit 200 --nemesis standard

Nemesis "standard" is a set of nemeses: kill, pause, partition and clock. All of them applied randomly during test.

Actual behavior

Amount of money must be always positive, withdraw cannot happen when amount of money is negative, see Lua code that implements SQL comment to transfer money between accounts https://github.com/tarantool/jepsen.tarantool/blob/master/resources/tarantool/jepsen.lua#L133-L155

Expected behavior

Sum of amounts money of all accounts is equal to 100, amount money on all accounts is positive.

Logs: logs.zip
Source code: https://github.com/tarantool/jepsen.tarantool/blob/master/src/tarantool/bank.clj#L74-L139
Version of test: tarantool/jepsen.tarantool@6165e6e

[sergos]

in jepsen.log there's following:

 #object[jepsen.os.ubuntu.Ubuntu 0x543c887 "jepsen.os.ubuntu.Ubuntu@543c887"],
 :engine "memtx",
 :time-limit 200,
 :version "2.8",
 :mvcc false,
 :workload :bank-multitable-lua,
 :accounts [0 1 2 3 4 5 6 7 8 9]}

Where MVCC is set to false. Hence in the mentioned test code

             box.begin()
               local bal_from = space_from:get(0).BALANCE - amount
               local bal_to = space_to:get(0).BALANCE + amount
               if bal_from < 0 or bal_to < 0 then
                 return false
               end
               space_from:update(0, {{'-', 'BALANCE', amount}})
               space_to:update(0, {{'+', 'BALANCE', amount}})
             box.commit()

there's no guarantee that the data will be the same for the bal_from = space_from:get and the space_from:update.

I believe it's a mistake in the test configuration. Please, double check it passess with MVCC turned on.

[ligurio]

everything fine without nemesis and enabled mvcc:

$ lein run test --workload bank-multitable-lua --nodes-file nodes --username ubuntu --version 2.9 --leave-db-running --mvcc
...
 {:SI
  {:valid? true,
   :read-count 923,
   :error-count 0,
   :first-error nil,
   :errors {}},
  :plot {:valid? true},
  :valid? true},
 :valid? true}


Everything looks good! ヽ(‘ー`)ノ

let's add nemeses: $ lein run test --workload bank-multitable-lua --nodes-file nodes --username ubuntu --version 2.9 --leave-db-running --mvcc --nemesis standard

Jepsen reports that sometimes total is wrong (not equal to 100):

 {:SI
  {:valid? false,
   :read-count 788,
   :error-count 330,
   :first-error
   {:type :wrong-total,
    :total 47,
    :op
    {:type :ok,
     :f :read,
     :process 8,
     :time 60932807892,
     :value {2 5, 3 0, 4 20, 7 19, 9 3},
     :index 408}},
   :errors
   {:wrong-total
    {:count 330,
     :first
     {:type :wrong-total,
      :total 47,
      :op
      {:type :ok,
       :f :read,
       :process 8,
       :time 60932807892,
       :value {2 5, 3 0, 4 20, 7 19, 9 3},
       :index 408}},
     :worst
     {:type :wrong-total,
      :total 0,
      :op
      {:type :ok,
       :f :read,
       :process 16,
       :time 61840800947,
       :value {},
       :index 465}},
     :last
     {:type :wrong-total,
      :total 0,
      :op
      {:type :ok,
       :f :read,
       :process 271,
       :time 114091365026,
       :value {},
       :index 3145}},
     :lowest
     {:type :wrong-total,
      :total 0,
      :op
      {:type :ok,
       :f :read,
       :process 16,
       :time 61840800947,
       :value {},
       :index 465}},
     :highest
     {:type :wrong-total,
      :total 59,
      :op
      {:type :ok,
       :f :read,
       :process 82,
       :time 97395675872,
       :value {1 4, 3 19, 6 21, 7 14, 8 1, 9 0},
       :index 2221}}}}},

Full logs:
jepsen-mvcc.zip

Tarantool version:

ubuntu@ubuntu-basic-1-4-10gb-s:~$ tarantool --version
Tarantool 2.9.0-167-gb1ee4e59b
Target: Linux-x86_64-RelWithDebInfo
Build options: cmake . -DCMAKE_INSTALL_PREFIX=/usr -DENABLE_BACKTRACE=ON
Compiler: /usr/bin/cc /usr/lib/ccache/g++
C_FLAGS:-g -O2 -fdebug-prefix-map=/build/tarantool-2.9.0.167=. -specs=/usr/share/dpkg/no-pie-compile.specs -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -std=c11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-gnu-alignof-expression -fno-gnu89-inline -Wno-cast-function-type
CXX_FLAGS:-g -O2 -fdebug-prefix-map=/build/tarantool-2.9.0.167=. -specs=/usr/share/dpkg/no-pie-compile.specs -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fexceptions -funwind-tables -fno-omit-frame-pointer -fno-stack-protector -fno-common -fopenmp -msse2 -std=c++11 -Wall -Wextra -Wno-strict-aliasing -Wno-char-subscripts -Wno-format-truncation -Wno-invalid-offsetof -Wno-gnu-alignof-expression -Wno-cast-function-type

[Totktonada]

In my opinion, we should reduce it to some bash+lua test case. Moved to the Ecosystem Team backlog.

[Totktonada]

Moved out from the Ecosystem Team backlog to highlight on the triage meeting.

[sergos]

The mentioned reduction is possible after enabling of Jepsen implementation in Lua, that @ligurio started as pet project.