enhancement(observability): allow querying transform outputs on transform components

[sghall]

cc @leebenson

Small PR to add the transform outputs to transform components. Transforms can pipe into other transforms so in order to build a complete graph we need to make those available.

@leebenson we talked a little bit about making it easier to get a directed set of links. After going through this, what we have is actually not bad. You really just want to query the "output" connections on each component. My query to get all the nodes and links ended up being like (removing the metrics here)...

query NodesAndLinks {
  sources {
    name
    transforms {
      name
    }
    sinks {
      name
    }
  }
  transforms {
    name
    sinks {
      name
    }
    transforms {
      name
    }
  }
  sinks {
    name
  }
}

Which in JS you can process pretty efficiently (removing some details here on the metrics):

const createNodeAndLinks = (data: NodesAndLinksQuery) => {
  const nodes: GraphNodeData[] = [];
  const links: GraphLinkData[] = [];

  const { sources, transforms, sinks } = data;

  sources.forEach((source) => {
    nodes.push(createNodeData(source.name));

    source.transforms.forEach((transform) => {
      links.push({
        source: source.name,
        target: transform.name,
      });
    });
    source.sinks.forEach((sink) => {
      links.push({
        source: source.name,
        target: sink.name,
      });
    });
  });

  transforms.forEach((transform) => {
    nodes.push(createNodeData(transform.name));

    transform.transforms.forEach((nextTransform) => {
      links.push({
        source: transform.name,
        target: nextTransform.name,
      });
    });
    transform.sinks.forEach((sink) => {
      links.push({
        source: transform.name,
        target: sink.name,
      });
    });
  });

  sinks.forEach((sink) => {
    nodes.push(createNodeData(sink.name));
  });

  return {
    links,
    nodes,
  };
};

[leebenson]

Looks great 👍

Pushed an update for schema.json. After making changes to the API, you can run make build-graphql-schema to pick those up.

Do you fancy taking a stab at updating the client and adding a test? Not blocking, but might be a good exercise to re-use some of the existing patterns and get a feel for how it works in each area.

[sghall]

Thanks @leebenson! Yeah, I'll follow this one up with updating the client and adding the tests. Be good to go through that and start looking at some of the code there. Going to merge this one so we can get the vector playground updated (works off the nightlies).

[leebenson]

Awesome 👍