New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Composing public GraphQL APIs #490

Closed
KyleAMathews opened this Issue Sep 15, 2016 · 24 comments

Comments

Projects
None yet
@KyleAMathews

KyleAMathews commented Sep 15, 2016

A more relevant question now as of yesterday ;-)

So with Github or any other of what I'm assuming is the coming avalanche of public GraphQL APIs, it'd often be really handy if you could compose all or parts of a public schema with your private one.

For example, what if all of your users had a connected Github account and you could write queries like:

{
  viewer {
    githubViewer {
      repositories { edges { node { name, id } } }

  }
}

Or if there was a Wikipedia GraphQL API (PLEEAASSEEE someone build this!):

{
  viewer {
    hometown { // Jump into Wikipedia here
      population
    }
  }
}

Ideally there'd be some helper code that'd consume the introspection query and auto-construct the schema locally + provide helpers for bridging between schemas. Then there'd need to be some sort of AST slicer to pull out the part of the query that's for the remote API and then send it there to be resolved.

Thoughts? Doable? Someone already explored this further than I have? NPM package I missed ;-)?

/cc @taion who helped develop some of these ideas.

@taion

This comment has been minimized.

Show comment
Hide comment
@taion

taion Sep 15, 2016

Contributor

It does seem like this could live in userspace. Something like

gitHubUser: {
  type: remoteType(gitHubSchema, 'User'),
  resolve: (obj, args, context, info) => (
    resolveRemote(
      GITHUB_API_URL,
      fragmentName => `user($user: ${obj.gitHubId}) { ${fragment} }`,
      info,
    )
  ),
}

Perhaps?

Contributor

taion commented Sep 15, 2016

It does seem like this could live in userspace. Something like

gitHubUser: {
  type: remoteType(gitHubSchema, 'User'),
  resolve: (obj, args, context, info) => (
    resolveRemote(
      GITHUB_API_URL,
      fragmentName => `user($user: ${obj.gitHubId}) { ${fragment} }`,
      info,
    )
  ),
}

Perhaps?

@stubailo

This comment has been minimized.

Show comment
Hide comment
@stubailo

stubailo Sep 15, 2016

Contributor

I'd love to work on something like this. My first thought would be:

  1. Import schema with introspection
  2. Namespace all types, for example by prepending GitHub_ and Wikipedia_
  3. Write some code or a DSL that adds some fields/resolvers to existing types, to wire together the schema; for example:
extend GitHub_Repository {
  wikipediaPage: Wikipedia_Page
}
resolver: (root, args, context, info) => {
  return <something goes here>
}

I think the main help here would be a way to get the relevant fragment to send to the Wikipedia API from somewhere like info.

Contributor

stubailo commented Sep 15, 2016

I'd love to work on something like this. My first thought would be:

  1. Import schema with introspection
  2. Namespace all types, for example by prepending GitHub_ and Wikipedia_
  3. Write some code or a DSL that adds some fields/resolvers to existing types, to wire together the schema; for example:
extend GitHub_Repository {
  wikipediaPage: Wikipedia_Page
}
resolver: (root, args, context, info) => {
  return <something goes here>
}

I think the main help here would be a way to get the relevant fragment to send to the Wikipedia API from somewhere like info.

@taion

This comment has been minimized.

Show comment
Hide comment
@taion

taion Sep 15, 2016

Contributor

For introspection, I think it'd be important to do that as a build step rather than a runtime step. You don't want to be firing off an introspection query at runtime.

Contributor

taion commented Sep 15, 2016

For introspection, I think it'd be important to do that as a build step rather than a runtime step. You don't want to be firing off an introspection query at runtime.

@stubailo

This comment has been minimized.

Show comment
Hide comment
@stubailo

stubailo Sep 15, 2016

Contributor

Agreed!

Contributor

stubailo commented Sep 15, 2016

Agreed!

@nybble73

This comment has been minimized.

Show comment
Hide comment
@nybble73

nybble73 Sep 16, 2016

I've been messing around with extendSchema - but as with a few other ideas I've played with, it falls apart on interface naming (i.e. the two services both are relay compliant so implement the node interface, but extendSchema of the external services graphql and the actual services node aren't the same so it conflicts - even though the shape of them is identical).

nybble73 commented Sep 16, 2016

I've been messing around with extendSchema - but as with a few other ideas I've played with, it falls apart on interface naming (i.e. the two services both are relay compliant so implement the node interface, but extendSchema of the external services graphql and the actual services node aren't the same so it conflicts - even though the shape of them is identical).

@theorygeek

This comment has been minimized.

Show comment
Hide comment
@theorygeek

theorygeek Sep 16, 2016

One important consideration would be how you secure it? That is, if the query against GitHub runs with elevated privileges, you probably care about the contents of the fragment you're sending to the remote server.

Maybe you only import part of the remote schema, and exclude parts that shouldn't be accessible? Or do you look at the AST of the inner fragment and apply security before you execute the remote query or something?

Just don't want something like this:

{
  viewer {
    someRemoteApi {
      node(id: "some-object-i-shouldn't-be-able-to-access") {
        ... on SomeType { sensitiveField }
      }
    }
  }
}

theorygeek commented Sep 16, 2016

One important consideration would be how you secure it? That is, if the query against GitHub runs with elevated privileges, you probably care about the contents of the fragment you're sending to the remote server.

Maybe you only import part of the remote schema, and exclude parts that shouldn't be accessible? Or do you look at the AST of the inner fragment and apply security before you execute the remote query or something?

Just don't want something like this:

{
  viewer {
    someRemoteApi {
      node(id: "some-object-i-shouldn't-be-able-to-access") {
        ... on SomeType { sensitiveField }
      }
    }
  }
}
@pluma

This comment has been minimized.

Show comment
Hide comment
@pluma

pluma Sep 27, 2016

This gets a huge 👍 from me because it ties in with my work on GraphQL at ArangoDB. Basically ArangoDB allows developers to build data-centric microservices in JS that run directly in the database and expose their own HTTP APIs. We've supported GraphQL using graphql-sync (which is a wrapper around graphql-js without promises) since ArangoDB 2.8. The upcoming release (thanks to the great example of express-graphql) will make it even easier to expose GraphQL APIs from those microservices.

With REST APIs these services would typically be put behind a public-facing proxy that takes care of authentication, CORS and so on. There are even dedicated SaaS solutions for this like Kong.

With GraphQL there's currently no way to tie multiple GraphQL APIs together like this. IMO this currently doesn't make for a compelling story for GraphQL as a replacement for REST APIs in microservices.

The security concerns when composing GraphQL APIs would be almost exactly the same as when providing a credentialed proxy to a REST API. A simple whitelist or blacklist would suffice IMO (though even that seems non-trivial to implement).

pluma commented Sep 27, 2016

This gets a huge 👍 from me because it ties in with my work on GraphQL at ArangoDB. Basically ArangoDB allows developers to build data-centric microservices in JS that run directly in the database and expose their own HTTP APIs. We've supported GraphQL using graphql-sync (which is a wrapper around graphql-js without promises) since ArangoDB 2.8. The upcoming release (thanks to the great example of express-graphql) will make it even easier to expose GraphQL APIs from those microservices.

With REST APIs these services would typically be put behind a public-facing proxy that takes care of authentication, CORS and so on. There are even dedicated SaaS solutions for this like Kong.

With GraphQL there's currently no way to tie multiple GraphQL APIs together like this. IMO this currently doesn't make for a compelling story for GraphQL as a replacement for REST APIs in microservices.

The security concerns when composing GraphQL APIs would be almost exactly the same as when providing a credentialed proxy to a REST API. A simple whitelist or blacklist would suffice IMO (though even that seems non-trivial to implement).

@taion

This comment has been minimized.

Show comment
Hide comment
@taion

taion Sep 27, 2016

Contributor

I don't think exposing a number of GraphQL APIs internally is an effective way to build a unified public-facing GraphQL API when using microservices internally.

The missing piece conceptually is that, in an effective GraphQL API, it's useful for types to form a graph. Suppose you were building a simple blog-like structure with a "users" service and a "posts" service. It's not just the case that these can trivially live in a flat URL-like space – the User type should have a posts: [Post] field, but the Post type should have an author: User field.

Unlike with a REST API where perhaps the post service can expose an author_id field, an effective GraphQL schema should actually expose relationships – in which case you need some way to express what they are, and then you're halfway toward actually building out a GraphQL schema.

I think the good pattern here is connecting to federated services in a limited way – having a nice way to link to the GitHub data for a user is great.

The idea of a GraphQL reverse proxy like you'd use for REST is IMO not a good pattern, though. Just build an actual schema – the richness available there is one of the main benefits of using GraphQL.

Contributor

taion commented Sep 27, 2016

I don't think exposing a number of GraphQL APIs internally is an effective way to build a unified public-facing GraphQL API when using microservices internally.

The missing piece conceptually is that, in an effective GraphQL API, it's useful for types to form a graph. Suppose you were building a simple blog-like structure with a "users" service and a "posts" service. It's not just the case that these can trivially live in a flat URL-like space – the User type should have a posts: [Post] field, but the Post type should have an author: User field.

Unlike with a REST API where perhaps the post service can expose an author_id field, an effective GraphQL schema should actually expose relationships – in which case you need some way to express what they are, and then you're halfway toward actually building out a GraphQL schema.

I think the good pattern here is connecting to federated services in a limited way – having a nice way to link to the GitHub data for a user is great.

The idea of a GraphQL reverse proxy like you'd use for REST is IMO not a good pattern, though. Just build an actual schema – the richness available there is one of the main benefits of using GraphQL.

@nybble73

This comment has been minimized.

Show comment
Hide comment
@nybble73

nybble73 Sep 27, 2016

We're moving toward a federated graphql microservices system as well. I agree that this is not a trivial undertaking. But if our internal structure has specific guidelines on naming and global ids, it should be possible to federate to each service requests that they provide and have the central dispatching service understand how to hydrate the links between them.

nybble73 commented Sep 27, 2016

We're moving toward a federated graphql microservices system as well. I agree that this is not a trivial undertaking. But if our internal structure has specific guidelines on naming and global ids, it should be possible to federate to each service requests that they provide and have the central dispatching service understand how to hydrate the links between them.

@KyleAMathews

This comment has been minimized.

Show comment
Hide comment
@KyleAMathews

KyleAMathews Oct 7, 2016

Drupal's graphql support seems to be getting really good https://youtu.be/yhyoQwuSUWo

KyleAMathews commented Oct 7, 2016

Drupal's graphql support seems to be getting really good https://youtu.be/yhyoQwuSUWo

@nodkz

This comment has been minimized.

Show comment
Hide comment
@nodkz

nodkz Oct 26, 2016

Contributor

@KyleAMathews I'm trying implement what you want in https://github.com/nodkz/graphql-compose
graphql-compose is a tool for building graphql types. Still in progress (polish API). So I'm not ready to write docs for it right now.

On top of graphql-compose, I builded this plugins:

All these tools I'm using for our internal project. So for OSS I'm moving not so fast as I want. And compose plugins for 3rd party GraphQL Services and REST API planned at the beginning of next year.

With graphql-compose and its plugins, the building of a graphql schema on the server is not pain anymore. For me ;) Ridiculously redeem from ctrl+с, ctrl+v. Just see on example app for northwind data, where ~700LoC converted 8 mongoose models to schema with 123 graphql types [live demo] [source code]. Relay app example which works with this northwind graphql server.


PS. Some funny slides about the graphql difference between frontend and backend developers:

screen shot 2016-10-26 at 12 20 30

screen shot 2016-10-26 at 12 24 50

Contributor

nodkz commented Oct 26, 2016

@KyleAMathews I'm trying implement what you want in https://github.com/nodkz/graphql-compose
graphql-compose is a tool for building graphql types. Still in progress (polish API). So I'm not ready to write docs for it right now.

On top of graphql-compose, I builded this plugins:

All these tools I'm using for our internal project. So for OSS I'm moving not so fast as I want. And compose plugins for 3rd party GraphQL Services and REST API planned at the beginning of next year.

With graphql-compose and its plugins, the building of a graphql schema on the server is not pain anymore. For me ;) Ridiculously redeem from ctrl+с, ctrl+v. Just see on example app for northwind data, where ~700LoC converted 8 mongoose models to schema with 123 graphql types [live demo] [source code]. Relay app example which works with this northwind graphql server.


PS. Some funny slides about the graphql difference between frontend and backend developers:

screen shot 2016-10-26 at 12 20 30

screen shot 2016-10-26 at 12 24 50

@linonetwo

This comment has been minimized.

Show comment
Hide comment
@linonetwo

linonetwo Oct 26, 2016

hahahah that's why we sometimes needs to use client side graphql server @nodkz

linonetwo commented Oct 26, 2016

hahahah that's why we sometimes needs to use client side graphql server @nodkz

@linonetwo

This comment has been minimized.

Show comment
Hide comment
@linonetwo

linonetwo Nov 16, 2016

Any progress on this?

linonetwo commented Nov 16, 2016

Any progress on this?

@linonetwo

This comment has been minimized.

Show comment
Hide comment
@linonetwo

linonetwo Dec 10, 2016

@stubailo I have an idea: Using JSON-LD 's concepts.

The JSON-LD Expansion is for transforming local types to global types in schema.org, to make a single source of types (think about the single source of truth). And namespacing a remote schema is about doing something like JSON-LD Expansion.

And JSON-LD Compaction is for using types in schema.org. It will transform long URL-like schema.org types to a short type name. (See https://www.youtube.com/watch?v=Tm3fD89dqRE if you aren't clear about it) . In GraphQL, we can just use introspection to do that, which is costly. But in JSON-LD way, what we need is to expose a 「@content」 field. And using a remote type means using something from a type source likes schema.org. we can do costless cascading by this, since cascading GraphQL may need to use tons of remote types.

Just an idea inspired by #271

linonetwo commented Dec 10, 2016

@stubailo I have an idea: Using JSON-LD 's concepts.

The JSON-LD Expansion is for transforming local types to global types in schema.org, to make a single source of types (think about the single source of truth). And namespacing a remote schema is about doing something like JSON-LD Expansion.

And JSON-LD Compaction is for using types in schema.org. It will transform long URL-like schema.org types to a short type name. (See https://www.youtube.com/watch?v=Tm3fD89dqRE if you aren't clear about it) . In GraphQL, we can just use introspection to do that, which is costly. But in JSON-LD way, what we need is to expose a 「@content」 field. And using a remote type means using something from a type source likes schema.org. we can do costless cascading by this, since cascading GraphQL may need to use tons of remote types.

Just an idea inspired by #271

@linonetwo

This comment has been minimized.

Show comment
Hide comment
@linonetwo

linonetwo commented Dec 21, 2016

This project seems to be using a remote schema https://github.com/leancloud/leancloud-graphql/blob/master/schema.js#L44

@loganpowell

This comment has been minimized.

Show comment
Hide comment
@loganpowell

loganpowell May 17, 2017

I'm from the Census Bureau. This would be huge for open data.

loganpowell commented May 17, 2017

I'm from the Census Bureau. This would be huge for open data.

@linonetwo

This comment has been minimized.

Show comment
Hide comment
@linonetwo

linonetwo May 17, 2017

Now there are DBs using GraphQL in their API…How to deal with it?

linonetwo commented May 17, 2017

Now there are DBs using GraphQL in their API…How to deal with it?

@kevinsimper

This comment has been minimized.

Show comment
Hide comment
@kevinsimper

kevinsimper Jul 5, 2017

I think you would want to publish the schema as npm package that would contain the GraphQL schema for wikipedia or github and then just mount that in and write your custom resolver that was needed already :)

kevinsimper commented Jul 5, 2017

I think you would want to publish the schema as npm package that would contain the GraphQL schema for wikipedia or github and then just mount that in and write your custom resolver that was needed already :)

@linonetwo

This comment has been minimized.

Show comment
Hide comment
@linonetwo

linonetwo Aug 25, 2017

cnschema is considering using json-ld to compose multiple source of truth cnschema/cnschema#8

linonetwo commented Aug 25, 2017

cnschema is considering using json-ld to compose multiple source of truth cnschema/cnschema#8

@linonetwo

This comment has been minimized.

Show comment
Hide comment
@linonetwo

linonetwo Aug 26, 2017

A real use case: Neo4j now can export GraphQL API from Neo4j-GraphQL Extension, If there is a PostgreSQL exports another GraphQL API, composing them into a single source of GraphQL becomes a problem.

There is a discussion:
neo4j-graphql/neo4j-graphql#54

linonetwo commented Aug 26, 2017

A real use case: Neo4j now can export GraphQL API from Neo4j-GraphQL Extension, If there is a PostgreSQL exports another GraphQL API, composing them into a single source of GraphQL becomes a problem.

There is a discussion:
neo4j-graphql/neo4j-graphql#54

@loganpowell

This comment has been minimized.

Show comment
Hide comment
@loganpowell

This comment has been minimized.

Show comment
Hide comment
@loganpowell

loganpowell Nov 22, 2017

@KyleAMathews I'm also getting started with Gatsby! Cool project... Windows 10 is giving me some trouble tho (side-note). Love what you guys are doing!

loganpowell commented Nov 22, 2017

@KyleAMathews I'm also getting started with Gatsby! Cool project... Windows 10 is giving me some trouble tho (side-note). Love what you guys are doing!

@IvanGoncharov

This comment has been minimized.

Show comment
Hide comment
@IvanGoncharov

IvanGoncharov Aug 30, 2018

Collaborator

I don't see any actionable item for graphql-js in this issue + discussion is stale so I'm closing it.

Collaborator

IvanGoncharov commented Aug 30, 2018

I don't see any actionable item for graphql-js in this issue + discussion is stale so I'm closing it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment