Apollo Federation and Apollo Opentracing

[tvvignesh]

Hi. May I know how does this work with Apollo Federation in place where the Apollo Gateway is going to send requests to multiple different microservices behind the scenes? In such cases, how do we configure this - At the gateway level or service level? Any sample implementations behind this?

Thanks in advance.

[DanielMSchmidt]

I am unsure how apollo federation works, does it allow you to write the resolvers for the services yourself?

[tvvignesh]

@DanielMSchmidt There is a good blog post about it here: https://blog.apollographql.com/apollo-federation-f260cf525d21

It has a gateway layer which gets the GraphQL queries and then the gateway knows which portion of the query to send to which microservice. So, it splits the GQL query to multiple services where resolvers are written and then gets the aggregated response from all the services back to the gateway.

[DotSpy]

@tvvignesh Hello, looking for the same thing, did u manage to find solution?

[tvvignesh]

@DotSpy None out of the box as of now since Federation is pretty new. We have to manually instrument the services using open tracing and trace it using jaeger.

[DotSpy]

@tvvignesh i stuck at gateway, because i'm not used to JS, i would really appreciate if u can provide some pieces of code for it
Update: ohh my mistake thought u implemented it, i will try to do it, if smth will be done will let u know

[sysadmind]

I was able to get this to work by using a custom datasource and overriding the willSendRequest() call.

On the willSendRequest() method, you don't have the info parameter with the request, so I was able to set it on the Context object on the ApolloServer. I took the approach of getting the root span and returning that in the context so that it's always available and then in the willSendRequest(), I created a child span and applied the headers to the request parameter so that downstream Apollo servers get the headers set too.

[DotSpy]

For now u can use just https://github.com/open-telemetry/opentelemetry-js/tree/master/packages/opentelemetry-node

[nadilas]

@sysadmind can you share your willSendRequest implementation?

[DotSpy]

@nadilas i tried this solution, to use it u need not only custom GraphQLDataSource but also change sources of apollo server. For now i suggest u just to include opentelemetry-js

[nadilas]

@DotSpy thanks. I may oversimplify this, but wouldn’t setting the span Id in the outgoing header and setting that as a parent id on the root span solve the stitching issue?

[DotSpy]

@nadilas yeap, it will, but to do so you need receive parent id via context as @sysadmind told

[sysadmind]

Here is the willSendRequest that I was able to use. Note that you need to implement the context

  willSendRequest({ request, context }) {
    const tracer = opentracing.globalTracer();
    const networkSpan = tracer.startSpan('federated request', {
      childOf: context.rootSpan ? context.rootSpan : undefined,
    });
    let injectHeaders = {};
    tracer.inject(networkSpan, opentracing.FORMAT_HTTP_HEADERS, injectHeaders);
    for (const [key, value] of Object.entries(injectHeaders)) {
      request.http.headers.set(key, value);
    }
    return () => {
      networkSpan.finish();
    };
  }

[nadilas]

thanks for sharing @sysadmind. Let me check if I got this right.

You are using opentracing.globalTracer() because you have used opentracing.initGlobalTracer() with your own tracer implementation?

By "implement the context" you probably something tracer.extract in the context: ({ req }) => {} of the apollo-server I assume?

Edit: I just realized you meant in the gateway and not within the federated service - as it already extracts incoming headers.

One more thing, are you sure rootSpan is the correct name? I'm using https://github.com/DanielMSchmidt/zipkin-javascript-opentracing as the implementation and it looks like it's using childOf: https://github.com/DanielMSchmidt/zipkin-javascript-opentracing/blob/948858890c1a833d69edd90ca44b87e6ed1ac184/src/index.js#L96

[sysadmind]

1. Yes I have used opentracing.initGlobalTracer(myTracer) so that I can reference it with opentracing.globalTracer() instead of having to pass it through other means. This function also returns a noop tracer if not previously set, so things like tests should still work even without initializing a tracer.
2. Implementing the context: correct, here's a simple version:

context: ({ req }) => {
    var externalSpan =
        req && req.headers
        ? tracer.extract(opentracing.FORMAT_HTTP_HEADERS, req.headers)
        : undefined;
    return {
        rootSpan: externalSpan,
    };
},

3. Federation: Yes this all happens in the gateway so that the downstream federated server can extract this span
4. rootSapn: I updated my above code. It was a bad copy paste. My apologies. The intent was to set the childOf attribute to the rootSpan from the context.

[nadilas]

Thanks great summary! Sounds about right :-)

What I still can’t figure out is, how I can attach the federated service to the span of the field which actually initiated the call to the federated service. I haven’t tested this yet, but it will attach all federated calls to the root and hence show a different (faster) timeline than what actually happens, right? I somehow fear that without info on willSendRequest we would need to hook into the fieldWillResolve and overwrite the rootSpan with the one from Info.span so we can grab that particular field.

Now the only question is, what happens in race scenarios? Maybe a map on context (url => span) would solve this, if we had the url available.

[DotSpy]

@nadilas few weeks ago i did https://github.com/open-telemetry/opentelemetry-js/tree/master/packages/opentelemetry-propagator-jaeger , u can try to use it

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.