Logbook 4
A Logbook entry written by @tpierrain
Like the previous time, we started to discuss with Cyrille, and code after with Mendel.
Topic of the day was:
Hexadecimal?!? No. Hexagonal? for sure! For those that needed a recap on what the Hexagonal -or port and adapters- architecture is => click HERE(beware: shameless bragging!!!)
You may ask why here... Why are we implementing an hexagonal architecture around our current & emerging OO business domain model?
The reason why we need it now is because our next big deliverable is an acceptance test harness for our Smart Order Routing System (SOR). We need somehow to expose a public (contract) interface for this test harness console application to play with whatever SOR implementations we will provide afterwards (our simple OO version first, then an LMAX-like version, a RX-based one, an F# one, etc). We also need somehow to externalize our Market mockups (feed replayer, order routing market mock) so that they can be used by every SOR implementation. As you can see, it's kinda infrastructure time for us (vs. domain model code). And since we don't want to tight any technological thing into our SOR system, the Hexagonal Architecture seems the best fit to keep focusing on the "inside".
Ok then: let's get back to our mob thing:
After having whiteboard-ed our hexagon, we identified its 3 ports and adapters:
-
SmartOrderRouting API (investor-side)
-
MarketDataFeed API (markets-side)
-
OrderRouting API (markets-side)
We were trying then to figure out the form of the third one: the OrderRouting port/adapter. Should it be a protocol or an OO API as we started to implement since the beginning? Should we introduce a kind of common layer upon all the various kind of markets to handle and protect from race conditions in the way we received the notifications/information? ... Wait a minute! Did you just said "race condition"?!? (you may ask "why" here).
Yes, you should... (in fact it will help me if you do it right now...)
- "Why?!?"
What does it mean? Cyrille illustrated that to us through the whiteboard:
Every time a client (like our SOR) is sending an Order to a Market, there are many steps involved on the Market side:
-
The Order book of the Market is updated, and that triggers a potential Execution via the Market's Matching Engine
-
This Order(s) matching will then send an Order(s)Updated message (Message 1 on the diagram) to the OrderRouting Market API used by our SOR.
-
Depending on the Market, an optional Execution notification message (Message 1bis) may also be sent to the OrderRouting Market API used by our SOR.
-
In every case, this Order matching will also lead to a Trade generation on the market side (next step after the Execution). To do so, the Market have to collect some needed extra data to be sent as part of the TradeInfo message (Message 2). This TradeInfo message is sent to the connected OrderRouting Market API of the involved Market client (i.e. our SOR).
-
On the Market (data) Feed side of the considered Market, this new execution may lead to a LastExecutedPrice update notification (Message 3) forwarded to every MarketFeed API connected to this Market (i.e. not via the OrderRouting API this time).
-
But this new trade will impact more deeply the FeedEngine of the Market, that will also send a FeedUpdated notification (Message 4) to every connected APIs (once this new execution will have impacted all related prices, available quantities/depths for this Instrument, etc).
Sounds logical right? But in reality
For a given Market you may receive messages 1, 2, 3, 4, with another Market you will receive 1, 1bis, 3, 4, 2 ... etc. It is also worth-noting that our SOR solver is mainly interested by 3 and 4.
Anyway. Our SOR must somehow be able to cope with those Market discrepancies. The question Cyrille, Mendel and I have been asking was:
- WHERE to handle those functional race conditions?
And our answer so far:
- Somewhere part of our business logic, and not embedded in a kind of Market APIs wrapper.
'Cause we don't want a leaky abstraction layer hiding from us some of the good Markets information and capabilities...
Ok. Cyrille left us, and Mendel and I started to code.
Funny thing: so far we were talking about the OrderRouting and MarketFeed APIs, but in fact we started to work on the SmartOrderRouting public API on the Investor side.
Indeed, we started to build a C# raw (i.e. without any middleware nor HTTP involved) port, and an adapter to translate from a flat DTO world ... to our internal business logic (OO world).
We continued to build it in TDD mode, but we've been asking tons of questions during the elaboration of this new public API for the SOR: how to expose our events in a non-OO world? How to do it without creating functional race conditions (where you post an investor instruction, and the corresponding answers/events are raised before you had the opportunity to subscribe to them, etc.).
As of today, you can have a look at:
- Our new service/API interface (BTW, we created a new project for SOR interfaces)
but also to
Note: We didn't have much time to work on it (lot's of talking before, even if it was necessary). So don't be cruel with us ;-)
Well guys... I think this is enough for today's Logbook entry. Next session, with (almost) the entire crew will be much more interesting regarding this SOR hexagonalization topic.
To be continued thus...
A Logbook entry written by @tpierrain
Without Cyrille, but with Ozgur and Tomasz this time, Mendel and I started to explain what we did during the previous session.
In fact, it turned into to a larger discussion. Even more: into a usefull clarification of what hexagonal architecture (a.k.a. Ports and Adaptors) meant to every one of us.
Yeah, because it was obvious: Mendel and I weren't enough clear so that Tomasz and Ozgur can quickly grasp what we did the day before (and what we intended to achieve in the long run). Thus, we -as a mob- decided to take a pencil, and to go to the whiteboard for clarification.
Everyone then tried to explain to the others the kind of hexagonal architecture he was willing to implement. As you will see below, we really needed some clarifications ;-)
Drew something that I tried to reproduce here in order to better grasp what was in stakes (note: the original white board picture is also displayed at the end of this post):
What do we have here: one Market on the top-right corner, one Investor (i.e. user of our hexagonal SOR) on the bottom-left. Lollipop representation for interfaces. The center of the (hexagonal) SOR for the applicative and domain code. And the weighted border of the hexagon for the infrastructure code landscape.
Every port/adapter combo being drawn as a blue-dashed rectangle, they all expose whether:
- A public interface to the external investor users (mainly outside->inside interactions)
or
- An interface to sustain interactions with the Markets from the inside of the hexagon (mainly inside->outside interactions).
While Tomasz was drawing his diagram (slighly different as this one, I must admit), I was bothered by something in his model: all the ports/adapters were somehow connected to each others with lines. Like if they were consistent. As a whole.
According to me, there is no need to share anything between the external public API of the SOR (on the investor-side), and what the SOR needed to play with for its external interactions with the Markets. I don't even want to add a coupling between those 2 different kind of port/adaper blocks (the left ones and the right ones).
In fact, I didn't really understood what Tomasz was trying to explain with his diagram (but we will see why in few lines ;-). Anyway, this was the moment where I decided to take the pencil, and to draw the Mendel-and-I version of the (ports and adapters) hexagonal architecture. At least, that's what I thought at the time... ;-)
Let's discover this:
Reproduced here just below, my graphical version was trying to highlight the isolation between all those sets ot port/adapter.
Ok. All those port/adapter may share part(s) of the SOR business logic (at the center of the hexagon), but not even necessary.
Drawing my version had 2 consequences:
-
We all realized that we didn't get what Tomasz wanted to say with his diagram. Indeed for Tomasz, the lines on his diagram were only suggesting a kind of pipeline with some port/adapters. According to him, the interesting debate was much more related to the boundaries of our hexagon and where port/adapters instantiation & composition is done.
-
Funnier thing: Mendel and I just realized that we didn't share exactly the same view of what we built together ;-P
In particular regarding the SmartOrderRoutingService (i.e.: our first port/adapter for me; something slightly different for Mendel...) Nice transition to present:
Attached to the YAGNI principle, I was considering our SmartOrderRoutingService as the first port/adapter for our SOR (a raw C# port/adapter on the investor-side).
Mendel considered it as a kind of common layer that any investor port/adapter could use to access the SOR in a message friendly way. Given that the SOR currently funnels order instruction status upates via events on investor instruction and that means that the investor instruction can't really be used as a Message/DTO.
For testing/benching purposes this could be useful for us if we want to use the SOR in-process but still use it in a way that will share as much of the infrastructure with the ports/adapters as possible.
Graphically speaking, Mendel's proposal could be represented like this. With the SmartOrderRoutingService port/adapter represented as the blue dashed form on the left (with 2 different interfaces pluged-in):
After those clarifications, we continued to talk about which version of the hexagonal architecture to implement all together. In fact, we really couldn't manage to share the same vision of the technical solution to be implemented (every one of us probably still being stucked somehow in our own mental model ;-)
Not able to collectively agree on the solution? => we decided to focus -at least- on its objectives:
And in fact we quickly agreed on 3 points to be implemented, whatever how we will code it:
- Our SOR business logic must be fully encapsulated. Why? Because we need to be able to switch from one implementation (e.g. C# LMAX-like) to another (e.g. F#) without breaking the public API (the one to be used by our common acceptance test harness).
- As a good side-effect, this will allow every of our SOR technical implementation to freely evolve without breaking these (a kind of anti-corruption layer effect)
-
We need to expose a public interface (understand "contract") for the investors (DTO oriented, and not OO)
-
The first user of this public interface will be our upcoming acceptance test harness Important to have in mind in order to continue to respect the YAGNI principle.
Ok... Near 1 hour of lunch-whiteboard-discussion... and still no test/code written ;-(
Mendel then suggested that we got back to the code... and we all agreed on that ;-)
But first things first. We all decided to update our "next steps" backlog file (the equivalent of Kent Beck's tiny piece of paper while he's working). Why's that? Because we didn't have lots of remaining time to code today. Thus, it seemed important to secure our next session productivity (by saving us some time to re-synch on our objectives next time).
-
To introduce the concept of (financial) instrument (mandatory to support the concept of Market data feeds)
-
To introduce the notion of market data and the MarketData API & Port/Adapter (starting from our acceptance tests)
-
To introduce the OrderRouting API & corresponding port/adapter
-
To create our acceptance test harness (i.e. a console application generating a report after each runs session)
Once these next steps written within our backlog file, we started to code by introducing the notion of Instrument Identifier. But quite shortly:
Ding-dong! End of the session :-(
As previously said in this post, here is the picture of our original whiteboard (part of it being erased before we took this snapshot):
A Logbook entry written by @tpierrain
You think we were done with the Hexagonal Architecture chapter? Naaaah! ;-P
Not really surprising if we consider our recent discussion on twitter after my previous logbook entry (BTW, sorry for having spammed you in the process Alistair ;-)
Indeed, even if we started to work on our acceptance test harness for the first time, we quickly got back to a design discussion about:
- the nature of the ports and adapters to be built
- when and where to instantiate and plug them
- which kind of interactions to orchestrate with our business logic (at the center of our hexagon).
In fact, it was the very first time we were all together in a session (Ozgur, Cyrille, Mendel, Tomasz and I) and it was really important that we were all on the same page for the upcoming dev day(s).
Yeah, because the more we talk about it, the more I prefer the "Ports & Adapters" (new) name of the pattern. I think it makes us focus on what's really important here.
Isolating our business logic from any technological dependencies through ports and adapters is all that really matter after all.
After minutes of discussion and white board, we stopped to draw an hexagon (original idea from Tomasz), and we collectively led to the following picture:
What do we have here...
-
No more hexagonal representation, we replaced hexagons with rectangles
-
We introduced the notion of Host, responsible to instantiate and plug the ports and adapters to the business logic (represented here within the Engine rectangle)
-
A Port is responsible to support and make the link with a technology (here whether RESTFull HTTP, TIBCO messaging, raw Inproc C#, OrderRouting protocol, or MarketDataFeed API).
-
Its corresponding Adapter is responsible to map or adapt the DTOs or data structure from the technological side to the ones from our business logic side (and vice-versa).
-
Graphically speaking, we represented every Port & Adapter within the same rectangle or square.
-
We decided to clarify our code by suffixing every Port type by "Port"
-
We decided to highlight the entry point of our systems from the other (services) ports (because we really feel that there are 2 different kinds of Ports and Adapters: the ones related to the usage of the system, the others related to what the system needs to work)
-
Our SOR entry point (application level and not core domain level) should support a new ISOREntryPoint interface. Concretely speaking, this interface will be implemented by our existing SmartOrderRoutingEngine (interface to be renamed)
-
Concrete Ports should never be referenced by our business logic (which should only use interfaces). It means here that the OrderRoutingPort will be manipulated by our core business logic through the IOrderRoutingService interface. Same applies to the MarketDataPort that will implement a IMarketDataProviderService
-
We feel that there is a need for a kind of Session helper to be shared by all our Ports and Adapters on the left. Indeed, for every answer/event coming back from the business logic (i.e. through the ISOR...EntryPoint interface) we need them to be able to track/identify/filter which are the one related to their pending Investor Instructions.
We found nothing on that last point within the Ports and Adapter (or Hexagonal Architecture) literature, and we feel it would be interesting to clarify those implementation points to make people really using it.
We'll I think this is it for today. I really have the feeling that the entire mob now is on the same page regarding this Ports and Adapters chapter, and be able to code what will be necessary to our major next step: the SOR acceptance test harness (i.e. a console application running a SOR implementation with InProc Market mocks and a MarketData feed replayer in order to produce a latency/throughput report at the end of every runs).
I can't stand waiting for the next mob lunchbox session!