Playing Devil's Advocate: Pennies vs. Quarters

[ryandotclair]

Let me preface all of this with: I like the idea behind this project. Any time I like something, I try to poke holes into it because I believe knowing the weaknesses sooner, rather than later, will only make it stronger.

With that said, I wrote an article on metaparticle, playing the devil's advocate. I came up with three main arguments against it: leaky abstractions, pennies vs. quarters, and questioning the philosophical choice of this abstraction layer's location, aka separation of duties. @brendanburns suggested I take this discussion to github.

Because leaky abstractions is just a fact of life, and something that will have to be dealt with, and because I'm not sure if the philosophical debate belongs in here (if you think it should be, I can add it as a separate issue), I'll just focus on the pennies vs. quarters argument for now.

The pennies vs. quarters idea stems from the saying "would you rather be holding a 100 pennies or 4 quarters?". Namely, because the abstraction layer is in the code, managing that abstraction layer becomes many orders of magnitude harder than an centralized abstraction layer that sits outside of the application code. Specifically, I'm referring to three scenarios:

• Changes to the underlying infrastructure, especially changes that touch many applications, will require a lot of redeploy/recompiles of application code due to the tight coupling (also arguably chewing up needless cycles of the developer--but this leads into the separation of duties debate).
• Changes to the underlying packaging/clustering technologies (Docker and Kubernetes in this case) that could break the abstraction layer. This introduces a dependency matrix nightmare.
• Changes to the underlying packaging/clustering technologies that would add new feature/functionality that the abstraction layer could leverage... which requires updates to numerous libraries and redeploy/recompile of many applications.

I look for to the discussion below.

[brendandburns]

Thanks for the feedback, both here and in the article.

Here are my thoughts:

1. "Changes to the underlying infrastructure,...."
   Metaparticle is aiming to encapsulate/abstract at the level of a single microservice. I don't want to encapsulate larger than that. Given Kubernetes as an abstraction layer, changes to the Kubernetes surface and below, really shouldn't impact the application. Kubernetes provides the isolation between infrastructure and application.

Changes that happen outside of the application (e.g. a certificate rotation) should be accomplished without a recompile, if things are factored properly.

Put another way, perhaps Metaparticle is a forcing function. If you can't do the changes you want to your infrastructure without forcing developers to recompile their Metaparticle application. Perhaps your infrastructure isn't sufficiently decoupled...

2. "Changes to the underlying packaging/clustering technologies"
   Here, I think we have to rely on the abstraciton layers to enforce good forward/backward compatability. If we believe that Kubernetes is the "machine code" for distributed systems, then relying on stable Kubernetes APIs is like relying on stable x86 instructions. You don't really worry that upgrading from generation 7 to generation 8 in CPU is going to break your application. If Kubernetes is doing it's job, the same should be true.

Over time, the abstraction layer can choose to adopt new features (just like you adopt newer CPU features) but with an understanding that this is a slow process that explicitly prevents running in older environments, just like most programs now rely on advanced instructions that are missing from an old Pentium processor.

3. "Changes to the underlying packaging/clustering technologies"
   This is sort of the opposite of the above. Definitely adopting an abstraction layer makes it harder to use the raw power of the underlying system. We should think carefully about how we allow an end-user to escape from metaparticle and write in "assembly language" themselves. Though I'd argue that for many interpreted languages (hello Javascript, etc) jumping out to raw assembly is quite hard and it hasn't really impacted it's usage for 80% of the use cases.

I think Metaparticle is explicitly trying to target the 80% use case, not the 99% use case and as such advanced features are kind of explicitly a non-goal for the system...

Anyway, would love to keep talking about this. The basic analogy for all of this (i think) is that Kubernetes/containers are "machine code" for distributed systems, and Metaparticle is a compiler.

Best
--brendan

[spencergibb]

I'd like to chime in with a question, specifically coming from the java world. I'm a bit confused about the packaging. In a CI/CD pipeline, an executable jar would be created. In metaparticle, that jar would have to be executed to create the right output for k8s, correct? Wouldn't that be better suited to a maven or gradle plugin? I may be missing something important though.

[brendandburns]

I don't have any problem with the notion that in a CI/CD pipeline, there would be a plugin which reads the annotations out of the .jar file (or somesuch) and use a Gradle/Maven plugin to do the same things that executing the binary does.

My primary goal is to get the configuration information into the code, alongside the application itself.

I like the idea that "just running it" does the right thing because I think that's where most end-users start out. Building/running applications on their local machine. Metaparticle tries to take the same experience to running an application in the cloud.

I don't think that you can take a novice developer and tell them that a CI/CD system is a pre-requisite for running in the cloud, and I don't want all of the configuration information in a Maven XML file.

On the other hand, we do need to have a way to transition into a CI/CD system when that becomes the right thing to do.

As to how we do that? That's definitely an open question. I filed:

#49

to track the question.

[spencergibb]

Thanks for the prompt, well thought-out answer.

[ryandotclair]

@brendandburns thanks for the reply. Let's see here...

Metaparticle is aiming to encapsulate/abstract at the level of a single microservice. I don't want to encapsulate larger than that. Given Kubernetes as an abstraction layer, changes to the Kubernetes surface and below, really shouldn't impact the application. Kubernetes provides the isolation between infrastructure and application.

Understood on the scope. I think the root of my worry here is on any change done outside of MP that would cause a breakage (which very well might be unfounded--I know enough to be dangerous of MP, so not sure if this problem even exists). I suspect if there is, it would be in the LB/Service touchpoints... but I haven't looked into the linkage MP provides.

If a given change would cause a breakage, then to do it "right" you'd have to force the change through MP, which IMHO would be a failure on MP's part. Chicken Little scenario is a full redeploy of all apps. However, if MP is stateless--in the sense that it uses k8s as the single source of truth--then this very well might be an unfounded fear.

Put another way, perhaps Metaparticle is a forcing function. If you can't do the changes you want to your infrastructure without forcing developers to recompile their Metaparticle application. Perhaps your infrastructure isn't sufficiently decoupled...

+1,000 if done right (and a big reason why I like the idea of this project and it's potential). I think the biggest hurdle here is defining (for a better lack of a word) the contract between MP and the infrastructure. Which config data will be passed through, and how sticky will it be?

In other words, if ALL changes can be done outside of MP without breaking it, and MP is really just an onramp to get App's onto the cluster (with maybe some distribute-system-helpers like the locking/master election you have today), then that would be the perfect scenario in my mind.

There's of course the #49 problem but it's a start.

...relying on stable Kubernetes APIs...

A risk 👍. I haven't been around k8s long enough to know how stable it's been historically... How big is the risk here?

[stealthybox]

Understood on the scope. I think the root of my worry here is on any change done outside of MP that would cause a breakage (which very well might be unfounded--I know enough to be dangerous of MP, so not sure if this problem even exists). I suspect if there is, it would be in the LB/Service touchpoints... but I haven't looked into the linkage MP provides.

This doesn't seem very different to me from the tools we use to expose applications running on a single server today; We can manipulate ports, interfaces, routing, and iptables.

Some environments are setup to be very automatic and permissive -- others require intentional intervention. Either way, all programming languages have core libraries that can be used to configure the use of these I/O resources.

Similar analogies can probably be made for file systems.

I haven't been around k8s long enough to know how stable it's been historically... How big is the risk here?

API stability is a strong focus for kubernetes.