rethinking the architecture

[afeld]

For those not familiar with the CAP project, we are managing approval workflows for different groups in the executive branch, at the moment focusing on purchase card holders getting approvals from supervisors.

We have multiple clients needing approval workflows, but all are sending different information through different templates. One group, for example, is "forwarding" GSA Advantage carts (which copies the order information into the email), whereas another is (in the near term) going to be sending a work order as a spreadsheet. With the current architecture, the size of this codebase will grow linearly with the number of different groups (specifically, the number of types of workflows) we need to support, in terms of parsers, templates, etc.

We need to start splitting the generalized approval functionality from the client-specific functionality. More specifically, let's separate the email templates from the approval logic, so that C2 (or whatever we rename this central piece to) knows nothing about Navigator, carts, etc). The central piece would essentially be a state machine that

• Knows about approval groups/flows
• Sends notification emails
• Handles approvals/rejections
• Keeps an audit trail of activity

Options:

API

Many of 18F's other projects are service/API-oriented, having a central server which our clients can communicate with through REST.

Pros

• Improvements effective for everyone immediately
• Theoretically minimal setup burden for new clients
• Centralized place for reporting

Cons

• We have to be more careful about breaking changes
• Single point of failure
• Handling of attachments more difficult

Open questions

• How are the templates specified? We could:
  • Have an admin UI where they provide Liquid/Mustache templates, then have them POST the values to be filled in for each thing to be approved
  • Have them send the HTML to be included in the email in the POST

Gem/Engine

In this setup, each client would create a new Rails/Rack app that includes the common functionality as a gem.

Pros

• They have full control over their app
• The information is stored in their database, meaning they can debug/query the data more easily
• The shared component only upgrades when they explicitly tell it to

Cons

• Additional setup per client
• They have to manage their own infrastructure, including
  • Database
  • Email sending/handling
• The have to use Ruby
• They have to upgrade the shared component manually
• Engines/Railties aren't widely used...there may be a good reason

---

Questions? Thoughts? Alternative ideas?

[adelevie]

I would selfishly like you to go the Engine/Gem route since your newly-gained expertise would be useful for https://github.com/18f/answers. That said, it definitely adds some extra labor and complexity.

[RobSpectre]

I suspect that the constituency of clients that would consume this service that also have Rails currently in their stack is not super high. Delivering as a web service would make the service available to a much wider swath of consumers.

You're going to have the debt of backwards compatibility anyway - might as well incur that with the instrument that gives you a greater adoption likelihood.

[adelevie]

Why can't a Rails Engine/gem be a consumer of the API? Now you have two problems :)

[konklone]

An API-driven approach is what we're all about, and I like that a lot. But it seems weird to break this up to follow a API/REST strategy if the clients will still all need to use Ruby and Rails engines.

Also, the top part says the API core would handle sending notification emails, but you have "Email sending/handling" listed as something the clients would each need to do themselves.

[afeld]

Maybe I was unclear: the central component (be it a gem or an API) would handle the actual sending of emails in either case. Making C2 API-driven means the clients could implement template-building with whatever language they want.

[cpapazian]

I'm -1 on Rails Engines. I think they do a lousy job of splitting out concerns. Think of all the Rails projects that end up tightly coupled to Devise.

Given these options, I would vote for an API and a Rails client implemented as a Gem. The Gem acts as dogfood and a reference implementation.

Careful, though. REST and State Machines are known to not play well together. For example, if you expose your carts as resources, you don't want clients to be able to update their state directly (really, you probably don't want to allow any updates to cart via the API), because you want to control that. Actions (on carts, etc) can be created, and then a mess of complicated backend logic is executed to resolve cart state. It all starts to look very unRESTful ...

Why do you need to do this re-architecture now? How many different use cases do you have so far? Is the complexity really slowing you down? How long will it take for this refactor to pay off in terms of time saved?

I'd consider a more incremental route. Figure out your data model ... create that separation of concerns within Rails. You'll have more flexibility to experiment and avoid locking yourself in to bad decisions. Once you feel you have it nailed down, it'll be trivial to release a public API.

[afeld]

REST and State Machines are known to not play well together.

Yeah, the state changes would happen within C2. The primary use of the API would probably be to kick off approvals, and maybe check state.

Why do you need to do this re-architecture now? How many different use cases do you have so far?

Three, and hopefully growing.

Is the complexity really slowing you down?

No, but I worry that the longer we wait, the more difficult it will be to detangle.

How long will it take for this refactor to pay off in terms of time saved?

Hmm, hard to estimate. We already have an API for cart creation, so it's mostly a matter of deciding how the templates are specified/passed, and separating the client code. Nothing is blocking or slowing us down right now, but it's a matter of hedging future technical debt.

[cpapazian]

I guess my main point is that i don't think simply separating the code into multiple repos gets you any of the architectural things you seem to be after. All of the hard can almost certainly be done in place.

Nothing is blocking or slowing us down right now, but it's a matter of hedging future technical debt.

Given that these aren't pressing concerns (and there aren't any performance/scalability issues), it looks like the incremental route is the right way.

[afeld]

I guess my main point is that i don't think simply separating the code into multiple repos gets you any of the architectural things you seem to be after.

My main interest is enforcing separation of concerns, and increasing reusability. Nothing is stopping us from continuing to put everything in the one codebase, but it will get harder and harder to change that course. "Monolith" isn't quite the right term because it contains a bunch of loosely related things... maybe an "nolith", or a "mosh pit"?

[cpapazian]

I've always gone with "monorail" when we're talking about Rails apps.

Anyway, I think the monolith/monorail is underrated.

Splitting things out actually adds a ton of complexity ... the infrastructure orchestration problems suddenly become nontrivial, you can no longer just rely on the database to maintain consistency, and that's not even getting into the new abstractions that you have to build into the app to communicate with the other services.

Then there's the performance issues. That's usually the driver for splitting things out ... you remove a specific bottleneck from the app and implement it in a way that it has a different scaling factor. But, you still have to balance that with new problems like latency and failure modes. And of course, performance issues rarely crop up where you expect. To make a argument from performance, you have to measure everything and have the data to back it up.

So, I'm still a -1 to all of this. Building the API into your monorail (note: you can spin up extra instances that only handle API traffic without splitting out the rest of the code) for now seems like the best course of action. You can still practice good separation of concerns within the monorail and avoid a bunch of extra complexity.

[dlapiduz]

+1 on @cpapazian comment of keeping everything in one app as long as it can handle it.

@afeld is there something going on with the current architecture that prompted you to rethink the architecture?
It seems to me that we should collect user feedback before rebuilding things...

[konklone]

I think Service Oriented Architectures made up of smaller, more modular parts are good things. I think the web is moving away from monoliths, and that that's a good thing. This doesn't feel like modularity in the SAO sense, though, since it sounds like API clients would be pretty tightly coupled to the internal functioning and technology stack of the API server. I could be misunderstanding.

[dlapiduz]

@konklone SOA is great. It is what everyone is moving to and it makes things easier to scale and allows for proper separation of concerns. I just think that in some cases the overhead of having a atomized system is not worth it.

Particularly, in this case, I would not prioritize re architecting the app just for the sake for being more modern...

[edit]: would _not_ prioritize

[konklone]

@dlapiduz totally, 👍 to all that.

[ultrasaurus]

I agree with keep everything in one place until you really understand the use cases and exactly what the app will need to do. It's hard to achieve separation of concerns, when you keep changing what you are concerned about. Monorails are really easy to refactor and change, then split up when you know your app and API are delivering the right value.

Maybe you are already on it, but I would be investing in metrics, KPIs, maybe a dashboard that is visible to stakeholders whose users are consuming API (through other products) as well as your UI. May seem off-topic, but I've seen success metrics change architecture and, of course, user experience.

[afeld]

Closing as stale.