Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Do not merge] 2.0 with schema stitching #382

Merged
merged 4 commits into from Oct 3, 2017
Merged

Conversation

freiksenet
Copy link
Contributor

@freiksenet freiksenet commented Aug 15, 2017

This is a WIP take on schema-stiching and schema merging. This will be published to a pre-release tag. See src/test/testingSchemas.js for examples..

API

makeRemoteExecutableSchema({ schema: GraphQLSchema, fetcher: Fetcher }): GraphQLSchema

Given a GraphQL schema (can be a non-executable client schema made by buildClientSchema) and a Fetcher, produce a GraphQL Schema that routes all requests to the fetcher.

introspectSchema( fetcher: Fetcher, context?: {[key: string]: any} ): Promise<GraphQLSchema>

Use fetcher to build a client schema using introspection query. For easy of use of makeRemoteExecutableSchema. Provides a client schema, so a non-executable schema. Accepts optional second argument context, which is passed to the fetcher.

Fetcher

type Fetcher = (
  operation: {
    query: string;
    operationName?: string;
    variables?: { [key: string]: any };
    context?: { [key: string]: any };
  },
) => Promise<ExecutionResult>;

Usage with apollo-link

import { HttpLink, makePromise, execute } from 'apollo-link';

const link = new HttpLink({ uri: 'http://api.githunt.com/graphql' });
const fetcher = (operation) => makePromise(execute(link, operation));
const schema = makeRemoteExecutableSchema({
  schema: await introspectSchema(fetcher),
  fetcher,
});

Usage with apollo-fetch

import { createApolloFetch } from 'apollo-fetch';

const apolloFetch = createApolloFetch({ uri: 'http://api.githunt.com/graphql'});
const fetcher = ({ query, variables, operationName, context}) => apolloFetch({
  query, variables, operationName
});
const schema = makeRemoteExecutableSchema({
  schema: await introspectSchema(fetcher),
  fetcher,
});

Usage with a generic HTTP client (like node-fetch)

import fetch from 'node-fetch';

const fetcher = async ({ query, variables, operationName, context }) => {
  const fetchResult = fetch('http://api.githunt.com/graphql', {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({ query, variables, operationName })
  });
  return fetchResult.json();
};
const schema = makeRemoteExecutableSchema({
  schema: await introspectSchema(fetcher),
  fetcher,
});

mergeSchemas

mergeSchemas({
  schemas: Array<GraphQLSchema | string>,
  resolvers?: (mergeInfo: MergeInfo) => IResolvers,
  onTypeConflict?: (
    left: GraphQLNamedType,
    right: GraphQLNamedType
  ) => GraphQLNamedType
})

type MergeInfo = {
  delegate(
    operation: 'query' | 'mutation',
    rootFieldName: string,
    args: any,
    context: any,
    info: GraphQLResolveInfo
  ) => any
}

schemas

schemas can be both GraphQLSchema (but it has to be an executable schema) or strings. In case they are strings only extensions (extend type) will be used. Passing strings is useful to add fields to existing types to link schemas together.

resolvers

resolvers is an optional a function that takes one argument - mergeInfo and
returns resolvers in makeExecutableSchema format.

mergeInfo and delegate

mergeInfo currenty is an object with one propeprty - delegate.

delegate takes operation and root field names, together with GraphQL context
and resolve info, as well as arguments for the root field. It forwards query to
one of the merged schema and makes sure that only relevant fields are requested.

mergeInfo.delegate(
  'query',
  'propertyById',
  {
    id: parent.id,
  },
  context,
  info,
);

onTypeConflict

onTypeConflict lets you customize type resolving logic. Default logic is to
take the first encountered type of all the types with the same name. This
methods allows customization of this, for example by taking other type or
merging types together.

Example

import {
  makeRemoteExecutableSchema,
  introspectSchema,
  mergeSchemas,
} from 'graphql-tools';
import { HttpLink, execute, makePromise } from 'apollo-link';

async function makeMergedSchema() {
  // Create remote executable schemas
  const PropertyLink = new HttpLink({
    uri: 'https://v7l45qkw3.lp.gql.zone/graphql',
  });
  const PropertyFetcher = operation =>
    makePromise(execute(PropertyLink, operation));
  const PropertySchema = makeRemoteExecutableSchema({
    schema: await introspectSchema(PropertyFetcher),
    fetcher: PropertyFetcher,
  });

  const BookingLink = new HttpLink({
    uri: 'https://41p4j4309.lp.gql.zone/graphql',
  });
  const BookingFetcher = operation =>
    makePromise(execute(BookingLink, operation));
  const PropertySchema = makeRemoteExecutableSchema({
    schema: await introspectSchema(BookingFetcher),
    fetcher: BookingFetcher,
  });

  // A small string schema extensions to add links between schemas
  const LinkSchema = `
    extend type Booking {
      property: Property
    }

    extend type Property {
      bookings(limit: Int): [Booking]
    }
  `;

  // merge actual schema
  const mergedSchema = mergeSchemas({
    schemas: [PropertySchema, BookingSchema, LinkSchema],
    // Define resolvers manually for links
    resolvers: mergeInfo => ({
      Property: {
        bookings: {
          fragment: 'fragment PropertyFragment on Property { id }',
          resolve(parent, args, context, info) {
            return mergeInfo.delegate(
              'query',
              'bookingsByPropertyId',
              {
                propertyId: parent.id,
                limit: args.limit ? args.limit : null,
              },
              context,
              info,
            );
          },
        },
      },
      Booking: {
        property: {
          fragment: 'fragment BookingFragment on Booking { propertyId }',
          resolve(parent, args, context, info) {
            return mergeInfo.delegate(
              'query',
              'propertyById',
              {
                id: parent.propertyId,
              },
              context,
              info,
            );
          },
        },
      },
    }),
  });

  return mergedSchema;
}

Known issues

  • Unions and interfaces don't work on remote schemas
  • Unions and interfaces don't work with remote schemas unless __typename is passed. However in mergedSchema it is done already, just need to figure out how to port it.
  • New lower level API that will allow handling links by just defining resolvers
  • Better default solution for remote custom scalars, currently they break eg JSON.
  • Make sure that aliases work
  • Types that only implement interfaces are missing
  • Better error handling - handle non fatal errros, make sure errors propagate from sub schemas, optional imperative interface to deal with errors from delegate.
  • Optional arguments in root fields don't work as expected when using delegate (and are basically ignored)
  • Generate var names in a more sane way, currently we just do _${argName}.
  • Figuring out if we want to add namespacing transform for schema

Production ready checklist

  • Documentation
  • Example app(s)
  • Blog post / announcement
  • Get rid of lodash and other extra deps
  • Validate usage in production for several people
  • Move all tests to not use .to.not.be.undefined, really hard to debug. use .to.deep.equal for full response

@freiksenet freiksenet requested a review from stubailo Aug 15, 2017
@stubailo
Copy link
Contributor

@stubailo stubailo commented Aug 15, 2017

Made the tests pass by adding @types/lodash package!

One idea: What if instead of prefixing field names, we put them in a nested field? Like:

# current
query {
  Booking_bookingById {
    ...
  }
}

# proposed
query {
  Booking {
    bookingById { ... }
  }
}

@stubailo stubailo changed the title [WIP] First take on schema stiching [WIP] First take on schema stitching Aug 15, 2017
stubailo
stubailo previously requested changes Aug 15, 2017
Copy link
Contributor

@stubailo stubailo left a comment

Some initial comments, more to come


public getSchema(name: string): GraphQLSchema {
if (!this.schemas[name]) {
throw new Error(`No such type: ${name}`);
Copy link
Contributor

@stubailo stubailo Aug 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be "no such schema" - do we even need these getters though? Would it be easier to just make schemas public?

Copy link
Contributor Author

@freiksenet freiksenet Aug 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then it won't type check :)

Copy link
Contributor

@stubailo stubailo Aug 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait really? Why not?

return this.linksByType[name] || [];
}

public getLinkByAddress(typeName: string, link: string): SchemaLink {
Copy link
Contributor

@stubailo stubailo Aug 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe getLinkByFieldName?


export default function addSimpleRoutingResolvers(
schema: GraphQLSchema,
// prolly should be a fetcher function like (query) => Promise<result>
Copy link
Contributor

@stubailo stubailo Aug 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep - I'll want this for the demo. Might write it

Copy link
Contributor Author

@freiksenet freiksenet Aug 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed.

const queryFields = registry.query.getFields();
return fromPairs(
links.map(link => {
const [schemaName, field] = link.to.split('_');
Copy link
Contributor

@stubailo stubailo Aug 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I think it would be better if we took two separate arguments for schema and field name, instead of splitting them on _. This couples it to the serialization we choose in a different part of the code.

Copy link
Contributor Author

@freiksenet freiksenet Aug 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, whole namespacing related code is a bit of a hack and will be better once we figure out proper namespaces.

@timbotnik
Copy link

@timbotnik timbotnik commented Aug 15, 2017

RE: prefixing - my initial use-case is to merge an existing in-production schema with a new one. This means that actually I don't want to prefix my existing schema at all, that should be supported shouldn't it?

@stubailo
Copy link
Contributor

@stubailo stubailo commented Aug 16, 2017

@timbotnik say more? Oh you mean you just want to graft a new one onto an existing one?

I think it would make sense to be able to have no prefix at all.

@timbotnik
Copy link

@timbotnik timbotnik commented Aug 16, 2017

@stubailo yes "grafting" is a good word for it ;-) Also, if you knew for sure that there would be no namespace conflicts, it could be useful to support a non-prefixed merge in general.

Thinking through a general case for non-prefixed merges, something that would be amazing would be if there was a way to handle de-duplication. I'm thinking more here about things like custom scalars and "utility" types that could be imagined to resolve exactly the same way on either endpoint.

Maybe more complicated, but also worth considering that an interface type might be shared between two schemas but each schema provides a different implementation.

@stubailo
Copy link
Contributor

@stubailo stubailo commented Aug 16, 2017

Yeah - I think the deduplication is probably going to be a future thing. Do you think it's critical for your initial use case?

@timbotnik
Copy link

@timbotnik timbotnik commented Aug 16, 2017

@stubailo I don't think it's critical at all, however the "custom scalar" thing is probably the thing that will look the strangest - like having Timestamp and Void [NS]_Timestamp and [NS]_Void in the final schema.

@stubailo
Copy link
Contributor

@stubailo stubailo commented Aug 16, 2017

Yeah, that's a great point. We should definitely work towards that. Having a concrete production use case would be super great.

@freiksenet
Copy link
Contributor Author

@freiksenet freiksenet commented Aug 16, 2017

@timbotnik We didn't really plan for "grafting" use case, some of the code would probably be shared, but I don't think we will support it in first iteration. However it seems that there are lots of use cases in schema merging, so I'll definitely think on how to allow some customization of how stuff will be merged, maybe providing functions to resolve type conflicts.

(Having said that, we really don't know what is the end goal of what we are doing, it's very much explanatory thing and a way to collect schema use cases. Maybe grafting is a very common one and then we'll certainly support it)

@freiksenet
Copy link
Contributor Author

@freiksenet freiksenet commented Aug 16, 2017

@timbotnik Re: de-duplication, our longer term plan involved having namespaces that can depend on other namespaces for shared types. Thus you could have 'common types' as one namespace and use scalars and shared types from there.

@freiksenet
Copy link
Contributor Author

@freiksenet freiksenet commented Aug 16, 2017

@stubailo This could work for field namespacing, but we would still need namespaces for types. I'm not sure :)

package.json Outdated
@@ -49,24 +49,26 @@
"homepage": "https://github.com/apollostack/graphql-tools#readme",
"dependencies": {
"deprecated-decorator": "^0.1.6",
"lodash": "^4.17.4",
Copy link
Contributor

@stubailo stubailo Aug 16, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmmm for the final release it would be super awesome to not include lodash - for client side use that really kills the bundle size.

Copy link
Contributor Author

@freiksenet freiksenet Aug 17, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we can either inline them or use lodash/function modules.

@timbotnik
Copy link

@timbotnik timbotnik commented Aug 17, 2017

@freiksenet I think perhaps the definition of "grafting" is getting confused. By this, all I mean is that I want to use an existing "local" schema (that I already have clients using in production), and merge a new schema onto it. Since the existing schema is already in use in the wild, I don't want to introduce a prefix on that schema. I think the only feature ask here is to allow me to turn off prefixing for this one schema during the merge.

@stubailo stubailo changed the title [WIP] First take on schema stitching [Do not merge] 2.0 with schema stitching Aug 17, 2017
@stubailo
Copy link
Contributor

@stubailo stubailo commented Aug 17, 2017

Just published 2.0.0-alpha.1! You can install with npm install graphql-tools@next!

@freiksenet
Copy link
Contributor Author

@freiksenet freiksenet commented Aug 17, 2017

@trevordmiller
Copy link

@trevordmiller trevordmiller commented Aug 18, 2017

I'm excited for makeRemoteExecutableSchema! Thank you.

@stubailo
Copy link
Contributor

@stubailo stubailo commented Aug 18, 2017

BTW here's my sample code from the talk at GraphQL NYC! This only works with 2.0.0-alpha.5 - the API for future alphas is a bit different.

import * as express from "express";
import * as bodyParser from "body-parser";
import { graphqlExpress, graphiqlExpress } from "apollo-server-express";
import { makeRemoteExecutableSchema, mergeSchemas } from "graphql-tools";
import { createApolloFetch } from "apollo-fetch";

async function run() {
  // XXX fix typings
  const universeSchema = await makeRemoteExecutableSchema(
    createApolloFetch({
      uri: "https://www.universe.com/graphql/beta"
    }) as any
  );

  const weatherSchema = await makeRemoteExecutableSchema(
    createApolloFetch({
      uri: "https://5rrx10z19.lp.gql.zone/graphql"
    }) as any
  );

  const schema = mergeSchemas({
    schemas: [
      { prefix: "universe", schema: universeSchema },
      { prefix: "weather", schema: weatherSchema }
    ],
    links: [
      {
        name: "location",
        from: "Event",
        to: "weather_location",
        resolveArgs: parent => ({ place: parent.cityName }),
        fragment: `
          fragment WeatherLocationArgs on Event {
            cityName
          }
        `
      }
    ]
  });

  const app = express();

  app.use("/graphql", bodyParser.json(), graphqlExpress({ schema }));

  app.use(
    "/graphiql",
    graphiqlExpress({
      endpointURL: "/graphql"
    })
  );

  app.listen(3000);
  console.log("Listening!");
}

try {
  run();
} catch (e) {
  console.log(e, e.message, e.stack);
}

Here's the query and result:

image

Also here are my slides for the talk: https://www.slideshare.net/sashko1/modular-graphql-with-schema-stitching

@stubailo
Copy link
Contributor

@stubailo stubailo commented Aug 19, 2017

I added tests that check for local schema merging, remote schema merging, and a hybrid of the two.

Should help us catch bugs with remote schemas! Already found one :]

@nnance
Copy link

@nnance nnance commented Aug 19, 2017

@freiksenet @stubailo this is amazing stuff. so much progress in such a short amount of time. so I have been very busy decomposing our production graphql server over the last couple of weeks. I have decomposed them into something we call GraphQL modules. It's in the perfect state to use this project to test a production use case. The only problem is that I will not be able to share the project. However, we are currently merging/stitching 6 modules together at build time to make up our GraphQL server including a pretty large set of common types so it should be a good use case. I will be able to share snippets as I find problems.

Would this be helpful?

@stubailo
Copy link
Contributor

@stubailo stubailo commented Aug 20, 2017

Absolutely! Please let us know whatever problems you run into, and also what features are missing for you to be able to use the tool :)

Super excited to see people getting pumped about this.

}

const resolvers: {
Query: ResolverMap;
Copy link

@timbotnik timbotnik Aug 21, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remote schemas are not working if they have root types that are not called Query or Mutation, such as

schema {
  query: RootQuery
  mutation: RootMutation
}

b/c the schema actually needs to be transformed as well to replace RootQuery => Query etc

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You get errors like Error: "Query" defined in resolvers, but not in schema

Copy link
Contributor Author

@freiksenet freiksenet Aug 22, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed that.

@gilesbradshaw
Copy link

@gilesbradshaw gilesbradshaw commented Aug 23, 2017

How easy to get this working over websockets?

@stubailo
Copy link
Contributor

@stubailo stubailo commented Aug 23, 2017

@gilesbradshaw you could easily get queries and mutations over websockets working with the right fetcher passed in, but for subscriptions it gets a bit more complicated for sure. That would be a big extra feature on top of this.

stubailo
stubailo previously requested changes Oct 3, 2017
Copy link
Contributor

@stubailo stubailo left a comment

This needs a changelog entry 😛

@freiksenet
Copy link
Contributor Author

@freiksenet freiksenet commented Oct 3, 2017

@sashko 🖕

This is ready to ship now :) :shipit:

@freiksenet freiksenet merged commit a834411 into master Oct 3, 2017
3 of 4 checks passed
@freiksenet freiksenet deleted the schema-stitching branch Oct 3, 2017
@lifeiscontent
Copy link

@lifeiscontent lifeiscontent commented Oct 3, 2017

@freiksenet where can I find the documentation for this now that this is merged?

@pozylon
Copy link
Contributor

@pozylon pozylon commented Oct 3, 2017

Awesome this seems to be done, but I just tried it and aliases don't work anymore if routed through makeRemoteExecutableSchema.

@martijnwalraven
Copy link
Contributor

@martijnwalraven martijnwalraven commented Oct 3, 2017

@pozylon: Can you elaborate on that and maybe include a code example that shows what doesn't work?

@pozylon
Copy link
Contributor

@pozylon pozylon commented Oct 3, 2017

Of course, sorry, here you go with the query:

{
  orders {
    status
    created
    updated
    ordered
    confirmed
    fullfilled
    currency
    it: total(category: ITEMS) {
      amount
      currency
    }
  }
}

I run it in GraphiQL and that was the result without proxying:

{
  "data": {
    "orders": [
      {
        "status": "CONFIRMED",
        "created": 1506586141020,
        "updated": 1506586183279,
        "ordered": 1506586183294,
        "confirmed": 1506586183294,
        "fullfilled": null,
        "currency": "CHF",
        "it": {
          "amount": 5958,
          "currency": "CHF"
        }
      },...

And that's what happens with proxying through the stitching:

{
  "data": {
    "orders": [
      {
        "status": "CONFIRMED",
        "created": 1506586141020,
        "updated": 1506586183279,
        "ordered": 1506586183294,
        "confirmed": 1506586183294,
        "fullfilled": null,
        "currency": "CHF",
        "it": null
      },...

total is of type "Money" which has a currency and amount (String + Int)

@martijnwalraven
Copy link
Contributor

@martijnwalraven martijnwalraven commented Oct 3, 2017

@pozylon: So are you saying it works if you remove the it alias for total?

@pozylon
Copy link
Contributor

@pozylon pozylon commented Oct 3, 2017

@martijnwalraven: exactly, if i call it without the alias it's all fine.

@martijnwalraven
Copy link
Contributor

@martijnwalraven martijnwalraven commented Oct 3, 2017

@pozylon: Great, thanks for explaining. @freiksenet, do you have any idea what could be causing this?

@gviligvili
Copy link

@gviligvili gviligvili commented Oct 4, 2017

I have a micro service (A) which has no protection.
so I stitch it to my main app server (B) which has protection.

is there a recipe on how to (B) can protect (A) queries? Like to if the user authorized or validate logic first?

@pozylon
Copy link
Contributor

@pozylon pozylon commented Oct 4, 2017

@gviligvili: Use a middleware together with apollo-fetch based Fetcher and cancel requests when appropriate. https://github.com/apollographql/apollo-fetch

@freiksenet
Copy link
Contributor Author

@freiksenet freiksenet commented Oct 4, 2017

@lifeiscontent we'll merge in the docs asap, see https://github.com/apollographql/tools-docs/pull/134 for progress.

@pozylon Thanks, i'll check it.

@freiksenet
Copy link
Contributor Author

@freiksenet freiksenet commented Oct 4, 2017

@pozylon Fixed in #411

@Techbinator
Copy link

@Techbinator Techbinator commented Oct 4, 2017

Hello again,

I have a small issue stitching a schema that was automatically generated with postgraphql library(https://github.com/postgraphql/postgraphql).

The service works fine on it's own but when i try to mergeSchemas i get "Unhandled promise rejection (rejection id: 1): Error: No such type: Query".

I already stitched 3 other services and it works perfectly.

You can find here the generated schema https://gist.github.com/filtudo/40ffe1bf27863d736e1646f9576dab4d

And here a link with the graphiql schema docs
https://pasteboard.co/GNmvbyz.png

Any help or guidance would be greatly appreciated.

Cheers

@freiksenet
Copy link
Contributor Author

@freiksenet freiksenet commented Oct 4, 2017

@filtudo The reason it doesn't work is because it refers to Query type in each payload. We handle Query type specially, tbh I didn't envision Query used this way. I wonder how common such use case is in practice.

Ping @sashko @jbaxleyiii @martijnwalraven

@Techbinator
Copy link

@Techbinator Techbinator commented Oct 4, 2017

@freiksenet

Any chance on "tackling" this issue on your side(since postgraphql i would say is a pretty decently used library) or i should start thinking of alternatives/hacks for this case?

Thank you

@freiksenet
Copy link
Contributor Author

@freiksenet freiksenet commented Oct 5, 2017

@filtudo It seems to be a common pattern in many APIs to support Relay, so I will fix it, yes.

@freiksenet
Copy link
Contributor Author

@freiksenet freiksenet commented Oct 5, 2017

@filtudo Fixed in #413

@freiksenet
Copy link
Contributor Author

@freiksenet freiksenet commented Oct 5, 2017

@filtudo Published as 2.2.1.

@pozylon Your bug is fixed in 2.2.0.

Hey folks, this is part of main release now, please create new issues if you have any problems with schema stitching, it's hard to track this thread cause it's closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet