Coder plugin: allows users to make custom Coder API calls

[Kira-Pilot]

Tasks (first added 2024-04-08)

From Michael: This is a harder issue than anticipated, so it's being turned into an umbrella issue with multiple pieces. Each bullet point will eventually become an issue – just need to fill everything out.

• Coder plugin: refactor codebase to use API factories #107
• Coder plugin: Ensure users always have way to add auth information even when not using Coder UI components #111
• Coder plugin: Make Coder SDK available to Backstage users #114
• Coder plugin: Add documentation for helping users cache state from CoderClient API calls #112

Original message

Right now, one is able to make calls to the Coder API via the Coder plugin; however, it is not as straightforward a process as it could be. We should expose a custom hook, fn, or other mechanism to facilitate ergonomic access to the Coder API through the Coder Backstage plugin.

Here is a thread with more background: #15 (review)

Desired Outcome

• An ergonomic API mechanism as described above
• Documentation within our Coder plugin README and API docs
• Load the token for the user if it's in local storage; set up authentication component as a standalone so the user can get the token if it's not already stored

[bpmct]

Primary discussion is here: #15

Edit: Here is the README copy to restore: 10eea7b

[Parkreiner]

Going to focus on this next week
My first order of business is finishing up and merging the refactor for our current API files (and also roll in #60, too). Once that's all done, I'll shift focus to adding support for all Coder endpoints

From what Kira and I were talking about at standup this week, the Coder Button is going to take a backseat until this is done. The way I see it, if we give really good API access, that would allow someone to make new buttons for whatever bespoke needs they have. That's going to be higher-impact than buttons that will (by necessity) only be designed for a handful of purposes

[Parkreiner]

Update on this: I started really diving into things yesterday, and I think that this is a thornier issue than I thought. It's going to need to be broken up into several issues. I've got a Notion document where I was brainstorming things, and thinking things through, so I can turn that into a set of GH issues soon

So far, I've been focused on refactoring the API file into more manageable pieces, and to start things off, I've been working on a refactor that uses API factories. I've been trying to refactor our auth logic into a separate API factory. The only thing is, I'm starting to bump into the technical limitations of these factories, and I don't know how viable they are yet. These things have a lot of pitfalls and edge cases around them when you're actually trying to use them for React state

I think they're mainly intended to be used as buckets of API functions, with no real state. Out of the box, this isn't enough for our auth needs, and I've been having trouble bridging the gap

I see one of two things happening:

1. We can successfully move all the auth logic into a separate file, and things are a lot better and easier to maintain than before
2. The technical limitations prove to be too much, and I have to throw the work away

Definitely going to have an answer by end of day, but I don't think there's any middle ground here. Still, even if I have to throw the auth refactoring work away, I have other refactoring done that makes the code better, and that I can keep. More importantly, I should still be able to refactor the other API logic into an API factory, and that should have two benefits:

1. Gets us closer to spinning up the Coder plugin in isolation for development
2. Will make it a lot easier to build out the custom hook for making any Coder request

[bpmct]

Thanks for the update. Looking forward to following along with the progress

[bpmct]

@Parkreiner I think I would be able to help most with reviewing / weighing the approaches if you provided sample docs/READMEs on what the developer experience is like using the API. e.g. docs-driven development.

[Parkreiner]

@bpmct Yeah, I can try to formalize what I've been working on so far.

Basically, the way that I had been thinking about the main end-user functionality is we'd have three "layers" of functionality for making API calls, and the user would be able to choose which one they're most comfortable with.

1. An unsafe API function that is the ultimate escape hatch (it lets the user hit any API endpoint with any arguments), but auto-complete is not great, and type-safety is even worse
2. A wrapper function that uses the unsafe API function under the hood, but provides much better type-hinting
3. A React hook that builds on the wrapper function by providing caching that gives the user a nice experience out of the box (this would use React Query under the hood). But it would be more opinionated and wouldn't allow for as much fine-grained control

It was set up this way because even if we only wanted to promote (3), we'd still need (1) and (2) as building blocks. I figured we might as well make those available, too. Right now, I'm working on (1). – (2) and (3) will need more work to make sure I've got the abstractions right.

All three would also have additional measures, like not letting API calls go through if the user isn't authenticated, and also automatically populating API calls with any necessary headers for working with the Coder proxy

The unsafe function (1) would look something like this:

export type ArbitraryApiCallFunctionConfig = Readonly<{
  endpoint: string;
  body: ReadonlyJsonValue;

  // Type definition is a TypeScript hack; the overall type definition collapses
  // down to type string, but the other methods specified still show up in
  // auto-complete
  method?: 'GET' | 'POST' | 'PUT' | 'DELETE' | (string & {});
}>;

declare function unsafeApiCall(
  config: ArbitraryApiCallFunctionConfig
) => Promise<ReadonlyJsonValue>;

function ExampleComponent () {
  // CoderClient is an API Factory, and will also have all the other safe API functions
  // we use for our main components
  const coderClient = useCoderClient();
  
  const onSubmit = (event) => {
    // Pretend we get data from the form via FormData object
    void coderClient.unsafeApiCall({
      endpoint: `/organizations/${organization}/members/${user}/workspaces`
      method: "POST",
      body: JSON.stringify({
        name: newWorkspaceName,
        // Other properties go here  
      })
    });
  };
    
  return (
    <form onSubmit={onSubmit>
      {/* Pretend the form has inputs for filling in necessary details */}
    </form>
   );
}

As for (2), I'm still trying to figure out if there the best way to make it type-safe. Ultimately, I'm trying to have something like this:

const makeWorkspaces = useCoderApiFunction(
  /* Inputs that make this function typesafe for making a new workspace */
);

const getAppearance = useCoderApiFunction(
  /* Inputs that make this function typesafe for getting appearance data */
);

// Support inputs for every other API endpoint we have

The only thing is that I feel like I'm running up against some of the limitations of TypeScript itself. I was hoping I could make a lookup object type that associates different method-endpoint combinations with specific inputs and outputs:

// This would not be exposed to the user and would be how things work under the hood
type CoderApiLookup = {
  GET: {
    "/buildInfo": {
      body: BuildInfoBody;
      output: BuildInfoReturnValue;
    };
    
    "/experiments": {
      body: ExperimentsBody;
      output: ExperimentsReturnValue;
    };
  },
  
  POST: { ... };
  PUT: { ... };
  DELETE: { ... };
}

This should actually work for the examples I have above – because the entire path name is statically defined. But we have a lot of dynamic endpoints, too, and while you can have a dynamic String type...

type CustomEndpoint = `/organizations/${string}/members/${string}/workspaces`;

TypeScript doesn't like those as object keys, and I'm figuring out if there's a good way to get around that. I have the beginnings of a solution, but (1) it needs to be cleaned up, and (2) I have no idea this is going to push the TypeScript type layer to its limits. It could cause VSCode's TypeScript LSP to crash if I fill in every single API endpoint

[Parkreiner]

I'm going to worry about (2) more once I have (1) working and stable, which brings me to where I am now. I've been spending the past three days refactoring all the API logic to use API factories – specifically, adding CoderClient and CoderTokenAuth factories.

I think I'm finally closing in on circling the square, and figuring out how to turn API factory data into UI state for good UX. The factories should solve a lot of problems:

1. Letting us run the Coder plugin in isolation
2. Making it really easy to add other API functions in the future
3. Making it easy to let the user choose whether they want token auth or the Coder OAuth when setting up the plugin
4. Cleaning up a lot of the current frontend code

I wish Backstage offered a more ergonomic way to make the API state usable in React right out of the box, but all the official solutions from the docs aren't enough for our needs

I didn't get a chance to fill out proper issues for everything I think we'll need yet, but here are the broad strokes:

• Refactoring the codebase to use API factories
• Adding the unsafe API function to the CoderClient factory
• Building non-intrusive UI components that can let the user insert their Coder auth token, even if they're not using any of our main components
• Bringing in the full Coder types through the SDK
• Building out a safer version of the API function hook that gives a good developer experience
• Updating tests and mock data as the above are completed
• (Eventually – will put on backburner as other issues get worked on) Make a hook for auto-caching the API data and exposing it as React state

Notion doc where I was thinking things through is here