Optimistic Locking friendly changes #86

paneq · 2017-08-29T16:48:21Z

It started here: https://github.com/RailsEventStore/aggregate_root/issues/8

3 possible ways of writing

Number

:none - works like -1 assumes stream empty so far, starts adding new in stream
given number : -1..Infinity - assumes N, starts adding N+1, N+2

Should work well for

non-legacy
event sourcing scenario
transaction around not required

Auto

:auto - assumes lock in higher layer. Will query for last N and start doing N+1, N+2... etc [New mode]
good for legacy
requires transaction and lock around

Any

:any - NULLable position in stream, order unknown
For copying into highly-contagious technical streams where we don't really care about exact position
Best-guess order can be determined based on EventInStream auto-increment id.

This is just a spike that I wanted to share with you.

What do you think @mpraglowski @pawelpacana @mlomnicki ?

Checklist

I collected the work from across all gems. https://github.com/RailsEventStore/aggregate_root/issues/8 It is missing required Travis changes to run rails_event_store_active_record on Mysql and Postgres

paneq · 2017-08-29T16:48:37Z

right now it is missing required Travis changes to run rails_event_store_active_record on Mysql and Postgres

I considered using `prepend` to initialize @unpublished_events = [] instead of using the lazy load pattern: def unpublished_events @unpublished_events ||= [] end but I decided the overhead is not worth it. Although we coud use it to set version to -1 as well. I will create a separate issue for it.

mlomnicki · 2017-08-30T07:57:23Z

rails_event_store_active_record/spec/spec_helper.rb

@@ -1,26 +1,55 @@
+if ENV['CODECLIMATE_REPO_TOKEN']


not needed anymore. We got rid of CC

mlomnicki · 2017-08-30T07:59:04Z

rails_event_store_active_record/spec/spec_helper.rb

-        t.text        :data,        null: false
+        t.integer     :position,    null: true
+        if ENV['DATABASE_URL'].start_with?("postgres")
+          t.references :event, null: false, type: :uuid


Do we actually need the :uuid? Maybe :string would suffice? With :string we don't have to enable pgcrypto

And what's wrong with pgcrypto?

Nothing wrong. It's not enabled by default. Are we going to force users to enable it even though it's not really needed?

mlomnicki · 2017-08-30T08:03:11Z

Excellent job @paneq! This is quite a big PR. Could we maybe split it into smaller chunks?

The checklist already contains some bits that could be extracted to separate PRs (mysql/postrgres, fix enrich_event_metadata, etc)

I would also open separate PRs for the following:

ability to publish multiple events
Event#hash
Discuss - do we have to depend on activerecord-import?

Makes sense?

gottfrois · 2017-08-30T07:58:17Z

rails_event_store_active_record/lib/rails_event_store_active_record/event_repository.rb

-      event
+    def append_to_stream(events, stream_name, expected_version)
+      events = [*events]
+      expected_version   = case expected_version


this should be in its own private method. It makes the append_to_stream very hard to reason about otherwise, also violates single responsibility principle

gottfrois · 2017-08-30T07:58:52Z

rails_event_store_active_record/lib/rails_event_store_active_record/event_repository.rb

+        when :none
+          -1
+        when :auto
+          eis = EventInStream.where(stream: stream_name).order("position DESC").first


we should use a repository to access this

it should also be in its own private method

The code we are writing here is the repository :)

it's the event repository, not EventInStream repository. I would make a clear distinction

gottfrois · 2017-08-30T08:00:25Z

rails_event_store_active_record/lib/rails_event_store_active_record/event_repository.rb

+          expected_version
+      end
+      in_stream = events.flat_map.with_index do |event, index|
+        position = if expected_version == :any


extract in its own method

gottfrois · 2017-08-30T08:01:01Z

rails_event_store_active_record/lib/rails_event_store_active_record/event_repository.rb

+        else
+          expected_version + index + 1
+        end
+        Event.create!(


should use the adapter

I don't see the point of this adapter anymore now that the repository needs to work with 2-3 ActiveRecord classes instead of 1.

i think it makes it way harder to re-write this repository for another ORM if they are a mixed of multiple active record classes instead of a single responsibility repository which maps to one active record class

gottfrois · 2017-08-30T08:01:42Z

rails_event_store_active_record/lib/rails_event_store_active_record/event_repository.rb

+      end
+      EventInStream.import(in_stream)
+      self
+    rescue ActiveRecord::RecordNotUnique


this class should know nothing about active record

Why would rails_event_store_active_record/event_repository.rb need to know nothing about active record when it actively uses active record to implement the needed features.

nvm, didn't see we are in active record namespace :)

gottfrois · 2017-08-30T08:02:10Z

rails_event_store_active_record/lib/rails_event_store_active_record/event_repository.rb

+          stream: "__global__",
+          position: nil,
+          event_id: event.event_id
+        )]


better indentation technique would be easier to read IMO

gottfrois · 2017-08-30T08:02:33Z

rails_event_store_active_record/lib/rails_event_store_active_record/event_repository.rb

+          metadata: event.metadata,
+          event_type: event.class,
+        )
+        [EventInStream.new(


should be extracted into its own method as well

gottfrois · 2017-08-30T08:02:46Z

rails_event_store_active_record/lib/rails_event_store_active_record/event_repository.rb

@@ -17,75 +54,75 @@ def delete_stream(stream_name)
    end

    def has_event?(event_id)
-      adapter.exists?(event_id: event_id)
+      Event.exists?(id: event_id)


should use the adapter

gottfrois · 2017-08-30T08:03:16Z

rails_event_store_active_record/lib/rails_event_store_active_record/event_repository.rb

    end

    def last_stream_event(stream_name)
-      build_event_entity(adapter.where(stream: stream_name).last)
+      build_event_entity(
+        EventInStream.preload(:event).where(stream: stream_name).order('position DESC, id DESC').first


should use the event in stream repository like the other comment earlier

gottfrois · 2017-08-30T08:03:48Z

rails_event_store_active_record/lib/rails_event_store_active_record/event_repository.rb

-      unless start_event_id.equal?(:head)
-        starting_event = adapter.find_by(event_id: start_event_id)
-        stream = stream.where('id > ?', starting_event)
+    def read_events_forward(stream_name, after_event_id, count)


same comments about adapters

paneq · 2017-08-30T08:13:50Z

The checklist already contains some bits that could be extracted to separate PRs

I agree. It's more so that I don't forget about those things than to actually include in this PR.

paneq · 2017-08-30T08:15:51Z

ability to publish multiple events

well... We could add such functionality in separate PR without releasing a new version because publishing multiple events using the old code is still going to be completely broken from optimistic locking perspective but maybe a separate PR for extending API is a good idea? Dunno

paneq · 2017-08-30T08:17:17Z

Event#hash

No brainer. Feel free to extract and merge. Very simple. Related mutant discussion https://twitter.com/pankowecki/status/902521997573455877

paneq · 2017-08-30T08:18:10Z

Discuss - do we have to depend on activerecord-import?

Ideally not, but I didn't want to write myself all the code for doing a single INSERT with multiple records. It might not be relevant if we migrate to transactions and 3 tables approach.

paneq · 2017-08-30T08:20:30Z

@gottfrois Thanks for the comments about extracting methods etc. This is right now however totally Work In Progress in which I am trying to find out if what we are trying to achieve is possible within the constraints that I described. I am still not exactly sure and there are bigger decisions to be made (3vs2 tables), handling concurrency etc. So once I am happy with the Big decisions, I am gonna focus on refactoring the smaller parts.

paneq · 2017-08-30T08:23:51Z

@gottfrois Also, I am trying to reach a state in which this can be merged to master, unreleased. And then everyone can easily improve the codebase with proper tests until we are satisfied with the results. And the we could do an official release. So I am gonna dismiss the review right now. But does not mean I don't agree. I just want to handle those kind of things later :)

gottfrois · 2017-08-30T08:24:32Z

no hard feelings :)

We will deal with those changes later :)

mlomnicki · 2017-08-30T08:40:27Z

Event#hash

No brainer.

Apparently mere developers such as me don't understand it :) I.e. how is this change related to optimistic locking? Why the big number is written in binary notation? Why do we even need to overwrite Event#hash, etc

paneq · 2017-08-30T08:46:27Z

Apparently mere developers such as me don't understand it :) I.e. how is this change related to optimistic locking?

In one race condition test I wanted to guarantee that all events are returned but the order is not known so I used Array#to_set and check if two sets are equal. That failed. Which brings as to your next question:

Why do we even need to overwrite Event#hash, etc

Because our Event has == operator which makes it behave like a Value Object (ignoring metadata). But value objects with are equal in == sense should also return the same hash which is used when you put such object into Hash or Set.

       expect({
         klass.new(event_id: "doh") => :YAY
       }[ klass.new(event_id: "doh") ]).to eq(:YAY)

       expect(Set.new([
         klass.new(event_id: "doh")
       ])).to eq(Set.new([klass.new(event_id: "doh")]))

These two are failing without implementing hash

Why the big number is written in binary notation?

It's easier that way to see how many bits it has.

mlomnicki · 2017-08-30T08:48:40Z

Wow that's a proper explanation. Much appreciated, thanks!!

Simple test to check if #has_event? passes if different string with same content still returns true.

This changes the original logic but it makes sense imho because we now have the concept of an event belonging to multiple streams.

We now have 2 (it can be later 3) classes responsible for storing events. I don't see much point in this adapter anymore.

paneq · 2017-08-30T17:45:18Z

Regarding mutations in rails_event_store_active_record... I don't think it makes sense to run them on postgres or mysql. Probably I should bring back Sqlite into testing suite and mutate only on that.

https://mutation-testing.slack.com/archives/C0A0RSER3/p1504110479000710 mbj/mutant#724

paneq · 2017-09-09T15:57:32Z

Today I realized that ideally I would want to come out with a DB schema that allows for Event Store as queue: ( #106 ) and since I don't want to keep the DB schema constantly I kind of coupled both problems together.

Kill lovely mutations.

I am not sure if `equal` will always work, but I think it will as two identical symbols must be the same object, no matter how constructed.

1. `preload()` only has observable perfromance effects so mutating it does not benefit and I am not sure how would we kill it anyway? I was thinking about adding in build_event_entity(record) a check like raise "use preload()" unless record.association(:event).loaded? but then how do we kill mutations around this ^^ check if does not external API in any way? 2. where('id < ?', before_event) automatically substitutes id so killing a few mutations

I was wrong in bc51e52 we can use nr of DB queries to verify preloading. It its not part of linted spec but additional test of a particular implementation.

I does not matter if we shift all postions by 1,2,3 etc. We use 1 so that first event in a stream is recorded with nr 0 :)

I had to remove_index :event_store_events_in_streams, [:stream, :position] because it was used by default when fetching the records and even without explicit order the order as defined by that used index. Surprise.

Conflicts: .travis.yml

paneq · 2017-09-25T16:52:04Z

@pawelpacana @mlomnicki before we merge this I have a few questions. Let's start with most important one:

add_index :event_store_events_in_streams, [:stream, :event_uuid], unique: true - Do you think we need it? I think it makes sense. Given event should appear in a stream once and only once. read_events_forward and other similar methods basically rely on that fact because they find position based on event id and if there were two events with same id in a stream then we could have random results and failures.

mlomnicki · 2017-09-26T08:44:45Z

@paneq yep, defo makes sense 👍

Conflicts: aggregate_root/spec/aggregate_root_spec.rb rails_event_store_active_record/Makefile rails_event_store_active_record/rails_event_store_active_record.gemspec ruby_event_store/Makefile ruby_event_store/lib/ruby_event_store/client.rb ruby_event_store/spec/subscription_spec.rb

Our codebase relies on non-duplicated events. Right now it is not possible to publish same event twice. But if we add append_to_stream, it will be possible to have same event in many streams. Don't mutate #detect_pkey_index_violated because it has logic which work on different databases and we can't run mutant across many DBs at the same time easily (we could have many connections but not worth it imho).

paneq · 2017-09-27T10:21:14Z

@mlomnicki @pawelpacana I want to merge today, please review holistically :)

The idea behind __global__ stream is that it contained all events but only once. It might not be needed right now but when we add the feature to copy event between streams it could be useful to have __global__ without duplicates.

So publishing in stream a would add event in stream a and __global__ and linking an event into stream b would only add it in stream b, without adding again to __global__. Does that make sense?

paneq · 2017-09-27T11:04:25Z

So far I was not sure how we could introduce a 3rd table streams with name and position and unique index on name that could work in :any mode and not have a race condition on creating the record. This is an actual problem we had in another system where 2 background jobs tried to write first event to the same stream.

mostlyobvious · 2017-10-10T22:25:43Z

ruby_event_store/lib/ruby_event_store/client.rb

@@ -87,7 +97,7 @@ def enrich_event_metadata(event)
      metadata[:timestamp] ||= clock.()
      metadata.merge!(metadata_proc.call || {}) if metadata_proc

-      event.class.new(event_id: event.event_id, metadata: metadata, data: event.data)
+      # event.class.new(event_id: event.event_id, metadata: metadata, data: event.data)


what's up with this comment?

Changing metatadata inside the instance vs new instance with new metadata. Keeping the old behavior was not possible during refactorings. A separate task would be to bring back the old behavior if still wanted, but test it properly. It was not.

mostlyobvious · 2017-10-10T22:32:26Z

ruby_event_store/lib/ruby_event_store/spec/event_repository_lint.rb

  end
-end
+
+  it 'reads events different uuid object but same content' do


I don't quite get the description about different uuid object

I wasn't clear at all here. we provide different string instance containing the same UUID. This makes it possible to establish whether we need == or eql? or equal? comparison logic. So instead of using the same variable, we use different string variable but with identical content.

Optimistic Locking friendly changes WIP

d1f52b8

I collected the work from across all gems. https://github.com/RailsEventStore/aggregate_root/issues/8 It is missing required Travis changes to run rails_event_store_active_record on Mysql and Postgres

paneq added 2 commits August 29, 2017 19:05

This will be useful later but not now

03227ab

paneq mentioned this pull request Aug 29, 2017

Consider using prepend module pattern to set constructor values in AggregateRoot #87

Closed

paneq added 2 commits August 30, 2017 09:03

Kill mutants

d092f83

Kill mutants

5c02bb9

mlomnicki reviewed Aug 30, 2017

View reviewed changes

gottfrois previously requested changes Aug 30, 2017

View reviewed changes

paneq added 7 commits August 30, 2017 18:32

Remove assignment as we don't map

ce469a0

kill mutant

4c9128d

Simple test to check if #has_event? passes if different string with same content still returns true.

Deleting a stream does not delete events in other streams

2b441ca

This changes the original logic but it makes sense imho because we now have the concept of an event belonging to multiple streams.

No point in this adapter

81cadc8

We now have 2 (it can be later 3) classes responsible for storing events. I don't see much point in this adapter anymore.

Fix tests to use UUIDs

a504c92

Test everything billion times

c8a0eeb

Fix migration template

5c5cba2

Ignore splat operator mutation due to a mutant bug

30565d3

https://mutation-testing.slack.com/archives/C0A0RSER3/p1504110479000710 mbj/mutant#724

mlomnicki mentioned this pull request Aug 31, 2017

Benchmarks for RES #100

Open

paneq added 15 commits September 11, 2017 20:28

Fix events chema in RES repo spec

6569e1f

Add lovely tests

c333720

Kill lovely mutations.

More strong check

aafbda2

I am not sure if `equal` will always work, but I think it will as two identical symbols must be the same object, no matter how constructed.

Verify preloading effect with nr of queries

90c661f

I was wrong in bc51e52 we can use nr of DB queries to verify preloading. It its not part of linted spec but additional test of a particular implementation.

Remove un-needed line

b65d92c

Use constant

ba24bfd

I does not matter if we shift all postions by 1,2,3 etc. We use 1 so that first event in a stream is recorded with nr 0 :)

preload for 1 event is not necessary

73ab114

No need for AR records

dfbe957

explicit sorting by position rather than accidental

b4c8d63

explicit sorting by position - working test

bc66797

I had to remove_index :event_store_events_in_streams, [:stream, :position] because it was used by default when fetching the records and even without explicit order the order as defined by that used index. Surprise.

One more explicit test

e5a1d64

Simplify statement

084680e

Merge remote-tracking branch 'origin/master' into locking_friendly

4248c46

Conflicts: .travis.yml

Merge remote-tracking branch 'origin/master' into locking_friendly

88afd76

paneq added 4 commits September 26, 2017 15:45

Handle EventDuplicatedInStream in InMemoryRepository

cde5ecc

Test separate instances

a9ca329

paneq merged commit b81796d into master Sep 27, 2017

paneq deleted the locking_friendly branch September 27, 2017 14:49

mostlyobvious reviewed Oct 10, 2017

View reviewed changes

Optimistic Locking friendly changes #86

Optimistic Locking friendly changes #86

Conversation

paneq commented Aug 29, 2017 • edited by mostlyobvious

3 possible ways of writing

Number

Auto

Any

Checklist

paneq commented Aug 29, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mlomnicki commented Aug 30, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paneq commented Aug 30, 2017

paneq commented Aug 30, 2017

paneq commented Aug 30, 2017

paneq commented Aug 30, 2017

paneq commented Aug 30, 2017

paneq commented Aug 30, 2017

gottfrois commented Aug 30, 2017

mlomnicki commented Aug 30, 2017

paneq commented Aug 30, 2017

mlomnicki commented Aug 30, 2017

paneq commented Aug 30, 2017

paneq commented Sep 9, 2017 • edited

paneq commented Sep 25, 2017

mlomnicki commented Sep 26, 2017

paneq commented Sep 27, 2017

paneq commented Sep 27, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

paneq commented Aug 29, 2017 •

edited by mostlyobvious

paneq commented Sep 9, 2017 •

edited