Expanding Support for Plugins

[Mcrich23]

This is a discussion building on #630, #633, and #717 as suggested by @jglogan here.

Overview

When building and discussing #239, it was requested that compose support actually come in the form of a plugin rather than build in. This did not seem like a problem, after all swift-argument-parser supports plugins by default. However, it quickly became abundantly clear that container was not set up with plugins in mind. This led to #603 and #635, which in total exposed command line tools to a programatic interface that could be imported instead of running the shell and reading out the difficult to parse data.

This was a massive leap in support for plugins, where it could leverage existing code instead of reconstructing commands manually. Unfortunately, one large problem was left unsolved: passing down flags. OptionGroups are a core part of container. They are used to standardize common flags and arguments and standardize a common understanding of behavior across commands. This is fantastic, but can become difficult for plugins to use.

After it became clear that #239 was not going to be merged, I began working on my own tool that imports container and leverages several commands inside of it to create a Docker Compose-like experience. As I was building it, I noticed that there were flags around downloading container images, logging, etc, that were part of these OptionGroups. I identified a potential future where users could expect the typical flags they can use in various commands to also work on plugins and developers could respect those flags, or just pass them down to programmatically called container commands. A flag pass through if you will.

While Container-Compose only currently uses the Logging OptionGroup, it initially to use more. When I was first implementing passthrough, I had to manually parse out a large number of variables into an array of strings that I could then pass with my actual arguments. This was a difficult and tedious process.

To make this more feasible, I created #717 and proposed that every flag OptionGroup adopt a new function called passThroughArguments that converts every variable in the structure to an array of strings that can then be concatenated with other arguments. This solution introduces a macro that dynamically generates the function at compile time and looks at every variable listed. This method of generating the function enables minimal changes to the current methodology of interfacing with ContainerCommands, while enabling pass through support for devs who want their plugins to respect user choices all the way down. It also is easy for future adoption of flags for passthrough support since plugins only need to be recompiled and new arguments will be added to the array.

Alternatives

At one point in time, there was a discussion of having properties of commands being public in addition to the parse static functions of structures in ContainerCommands. If we want to revisit that, exposing flags variables for commands would also remove the current roadblock in the container plugin ecosystem.

---

I would love to hear other's thoughts on this matter so I/we can build a solution that supports anyone and everyone interested in developing plugins in the future. Let's use this discussion as a chance to craft a more solid direction before any more work is done regarding this matter.

[Mcrich23]

@jglogan I am realizing one issue with both approaches which is version upgrading. Because all of the command behavior is compiled in the binary with one version of container, but the container CLI version has the potential to be different and create a bad state in the backend serving files between even the most minor of version changes. Put differently, if you have a PluginA that is built with ContainerCommands 0.8 and PluginB that is built with ContainerCommands 0.11, they may conflict in behavior and cause a bad configuration that even the official cli will not understand. This would leave that container in an unusable state.

To address this, I wonder if it may be better to create a shared xpc server for both plugins and the official cli to use that keeps behavior consistent and streams back data. In which case, we would even be able to make a separate super light weight container sdk/package for plugins to use without cloning all of the dependencies container requires.

[jglogan]

if you have a PluginA that is built with ContainerCommands 0.8 and PluginB that is built with ContainerCommands 0.11, they may conflict in behavior and cause a bad configuration

Under what conditions do you see any two plugins conflicting with one another?

Plugin A is a binary executable that interacts with container at the level of the container client API, using the 0.8 version of the XPC protocol.

Plugin B is a binary executable that interacts with container at the level of the container client API, using the 0.11 version of the XPC protocol.

If we maintain client API compatibility within major versions*, what is the issue?

* major version 0 will be shaky on this but I think we're good from 0.10.0.0 on and we should generally work from 0.8.0.0 on thanks to the work @saehejkang and @edsai did for #1196

[Mcrich23]

Never mind, it appears that my concerns on this come from outdated knowledge based on working with the very early versions of container. However, it is important to note that structures like HostDNSResolver would be susceptible to this issue, but I doubt it will ever have breaking changes between minor versions anyway.

Let's proceed with examining my initial proposal above and mostly disregard this comment.