Super Graph as a library (aka. embedded mode)

[dosco]

What would you like to be added:

A clean API to integrate Super Graph into other GoLang apps. This would be done as a http handler that can be plugged into an existing router or at a much lower level where you can provide a config object and create a new Super Graph instance to be used in your code.

Also hooks can be added for various things like onQuery, onMutation, onMutationComplete, etc, etc. This would help code using Super Graph as a library provide their own behaviour to execute during request handling.

The Super Graph GraphQL compilers (QCode and SQL) are already available as a library, this work would focus on moving more pieces of the serv package into a clean API.

Why is this needed:

Currently I run two services one for custom apis like authentication and the other Super Graph. Going ahead other custom endpoints like file upload etc would possibly also be added to the first service. It would be great if I could instead bundle it all together into a single app and also be able to augment Super Graph with my own app code.

[ansarizafar]

@dosco Thanks for this amazing project. I would like to encode query response with https://github.com/fxamacker/cbor Will it be possible?

[dosco]

Super Graph generates JSON using Postgres so if this cbor library can parse JSON then it would be possible. To switch out the encoding entirely to cbor would be very hard and introduce additional allocations and complexity.

[ansarizafar]

I want to use Supper Graph with Dart/Flutter and Its difficult to decode graphql query response in json to Dart types. this Dart package https://pub.dev/packages/cbor can decode cbor to dart types. It would be great If there is an option to send query response in cbor.

Super Graph is great and we can further make developers life easy by providing a data modeling/Auto migration and query builder feature like Prisma2 https://github.com/prisma/prisma2/blob/master/docs/data-modeling.md , Edgedb https://edgedb.com/roadmap or Facebook Ent https://edgedb.com/roadmap to make it a complete data access solution.

[dosco]

I'll look more info cbor when I get some time. It's surprising that Dart/Flutter has a hard time with JSON which is pretty much the standard.

Super Graph just uses the data model defined in the database I'm unclear on the value of duplicating that again on another file. Built-in support for DB migrations allows you to update and manage this data model having one more file to update can get out of sync and slow you down.

[ansarizafar]

Super Graph just uses the data model defined in the database I'm unclear on the value of duplicating that again on another file. Built-in support for DB migrations allows you to update and manage this data model having one more file to update can get out of sync and slow you down.

I wanted to say, It would be great If we can define data model in a schema file and not in the database and then Super Graph auto generate DDL and apply it to database. We can version control this file with our project code repo. Here is an example of a schema file from Prisma2 docs.

// schema.prisma

datasource sqlite {
  url      = "file:data.db"
  provider = "sqlite"
}

model User {
  id        Int      @id @default(autoincrement())
  email     String   @unique
  name      String?
  role      Role     @default(USER)
  posts     Post[]
  profile   Profile?
}

model Profile {
  id   Int    @id @default(autoincrement())
  user User
  bio  String
}

model Post {
  id         Int        @id @default(autoincrement())
  createdAt  DateTime   @default(now())
  updatedAt  DateTime   @updatedAt
  author     User
  title      String
  published  Boolean    @default(false)
  categories Category[]
}

model Category {
  id    Int    @id @default(autoincrement())
  name  String
  posts Post[]
}

enum Role {
  USER
  ADMIN
}

Super Graph has made developers life very easy by providing database access from client via Graphql but we also need to access database from server. Go is great but relational database access with SQL is really hard. I have yet to find a good solution. Super Graph can fill this gap If it can provide Prisma2 like query and migration functionally in addition to Graphql API.

I would request you to please check Prisma2 https://github.com/prisma/prisma2/blob/master/docs/data-modeling.md and EdgeDB https://edgedb.com/roadmap when you get time.

[ansarizafar]

Currently Super Graph is using Chai, Would it be possible to use any other router/framework like Fiber https://fiber.wiki/ or Echo https://echo.labstack.com/ ? Both frameworks are faster than Chai.

Here is how I would like to use amazing Super Graph;

Define database schema and access control in schema file

Super Graph can use this schema to create/update database tables without requiring users to create migrations

# database.schema
# Define schema with Graphql SDL (https://github.com/genie-team/graphql-genie)
	
enum Role {
	# Open to all requests
	ANY
	# Must be logged in
	USER
	# User must have created/be the type
	OWNER
	ADMIN
}

# Only users can create posts, anybody can read posts, only the person who created the post can update/delete it
type Post @partition(type: list, value: author) @auth(create: USER, read: ANY, update: OWNER, delete: OWNER) {
	id: ID! @unique
	title: String!
	text: String
	# Set a rule of "SELF" here that we can look for in our authenticate function so users aren't allowed to change it to other users
	author: User @relation(name: "posts") @auth(create: USER, read: ANY, update: OWNER, delete: OWNER, rules: "SELF")
}

# Anyone can create users (aka signup), be able to see user info by default, can only update yourself, and only admins can delete
type User @auth(create: ANY, read: ANY, update: OWNER, delete: ADMIN) {
	id: ID! @unique
	username: String! @unique
	# only users can see their own email, it's not public
	email: String! @unique @auth(create: ANY, read: OWNER, update: OWNER, delete: ADMIN)
	# only admins can read password
	password: String! @auth(create: ANY, read: ADMIN, update: OWNER, delete: ADMIN)
	posts: [Post] @relation(name: "posts")
	# Only admins can alter roles, will need additional logic in authenticate function so users can only set themself to USER role
	# So we set only:USER in the rules so we can find that later in our authenticate function
	roles: [Role] @default(value: "USER") @auth(create: ANY, read: ADMIN, update: ADMIN, delete: ADMIN, rules: "only:USER")
}

Use Super Graph as a module

package main

import "github.com/gofiber/fiber"
import "github.com/dosco/super-graph"
import "github.com/jackc/pgx/v4"

func main() {
  // Create new Fiber instance:
  app := fiber.New()
  db, _ := pgx.Connect(context.Background(), os.Getenv("DATABASE_URL"))
  defer db.Close(context.Background())

  graphql = super_graph.New()
  graphql.config.db = db
  graphql.schema_path = "./schema"
  graphql.config.xxx = // Other configurations

// Extend queries
graphql.extend.queries(// query definition)

// Extend mutations
graphql.extend.mutations(// Mutation definition)

// Allowed named queries
graphql.whitelist.add( `query getUserById($userId: Int) {
  user(id: 4) {
    id
    username
    email
  }
}`) 

  // Create route for graphql:
  app.post("/api", graphql.handler) 

  // Custom endpoint
   app.Get("/", func(c *fiber.Ctx) {
    // Query on server
   result , _ =    graphql.query(// Graphql named query)
    c.Send(result)
  
   })


  // Start server on "localhost" with port "8080":
  app.Listen(8080)
}

[frederikhors]

I can't wait for this!

[ansarizafar]

I have added a partition schema directive for declarative partitioning and query white listing to my proposal.

[frederikhors]

@ansarizafar is there an easy way to integrate it today in a Golang project?

[dosco]

@ansarizafar this is technically not hard to do expect for the reading schema stuff that does not exist so will have to be built, however I don't think we need it initially Super Graph does a good job automatically resolving schema and relationships directly from the database.

As for the part about adding and using named queries technically this is exactly how the Super Graph seed.js works. Not sure if you have seen this https://supergraph.dev/guide.html#seed-js

In the seed file you have a build-in function called graphql so you can use graphql queries to seed your database with say test data or whatever. This function even through you call it in Javascript is executed internally the Super Graph GO code and supports all of it's syntax query,mutations, batch inserts-updates, whatever. Look at serv/cmd_seed.go for the code there's a function called graphQLFunc(query string, data interface{} ...) all you have to do is expose this. Yes it won't do prepared statements but thats ok for now the compiler is very fast and adding that in is pretty easy once this initial stuff is done. Adding to an allow.list is not required we can just generate prepared statements on the fly.

  var res = graphql(" 
  mutation { 
    user(insert: $data) { 
      id
    } 
  }", { data: data })

Note 1 - Super Graph uses a GO JS interpreter to run the seed.js file as the graphql function returns JSON which is really easy to mold in Javascript.

Note 2 - For now only 1 instance of Super Graph can be created in your app since it makes use of some globals serv/cmd.go I don't think this is an issue but if it is then the solution would be to package those globals into a struct and make it the Super Graph context.

[ansarizafar]

this is technically not hard to do expect for the reading schema stuff that does not exist so will have to be built.

Let's Do It.

I don't think we need it initially Super Graph does a good job automatically resolving schema and relationships directly from the database.

Super Graph's goal is to help developers to "Build web products faster" Auto migrations will help achieve this goal. We can define complete data structure with data access rules on tables/field easily in one schema file without using SQL and creating a migration file after every change in data structure.

The suggested schema is just using Graphql SDL so there is no need to interscope database. The SDL is easily extensible with directives, we can build schema features like declarative partitioning etc gradually.

As for the part about adding and using named queries technically this is exactly how the Super Graph seed.js works.

Super Graph complies graphql queries to single SQL statement and that's the main advantage, no other ORM (Prisma,GORM, Facebook ant etc) except Micosoft Entity framework has the ability to generate single SQL statement for eager loading. I have recommended an API to use queries/mutations on the server so that we don't have to use SQL and database driver directly.

// Query on server
   result , _ =    graphql.query(// Graphql named query)

Adding to an allow.list is not required we can just generate prepared statements on the fly.

The suggested featured allow developers to add queries to a white list. The white listed queries should be complied to single prepared SQL statement in advance and only white listed queries should be allowed from Graphql endpoint in production.

// Allowed named queries
graphql.whitelist.add( `query getUserById($userId: Int) {
  user(id: 4) {
    id
    username
    email
  }
}`) 

Super Graph has a great potential to become a complete and de facto database access solution for developers all around the world but I would not like to use it in its current form as I don't like to use a separate standalone system with a separate config file (Prisma1 failed exactly of this reason).

I don't like to define actions/remote joins in a configuration file to extend the Graphql Api. The suggested Query/Mutation extend and query on server are far better features than the current solution.

In the end Super Graph is your project, you have done the hard work and its your right to decide what better suits your needs.

[dosco]

Super Graph is a community driven project and not just my project which is why we have these open discussions. As the only current core maintainer I'm more than happy to review and guide developers with any PR's they are working on. Yes it helps to come to some kind of consensus here through a discussion before a PR is worked on.

In short making Super Graph work as a library was always on the books this discussion thread was created as a way to gather more inputs to help with the design process. The sample code you submitted above to show how Super Graph as a module would work is really helpful so thanks for taking the trouble. It helps conceptualize how the API would look.

I agree there is no reason to force people to only use it as a standalone service when we can easily do both. As for auto-migrations I've also wanted this feature but personally have never seen it as yet used in real production systems. But I'm all for doing it as I rather maintain a GraphQL schema file instead of SQL migrations.

I'll start doing some initial looking into the Super Graph as a library thing sometime next week.

[ansarizafar]

It would be great If we can have an initial version without auto migration via schema file. We can later provide a cli to interscope a database to create a schema file for existing projects.

[frederikhors]

It would be great If we can have an initial version without auto migration via schema file. We can later provide a cli to interscope a database to create a schema file for existing projects.

Yes, these are two very separate things.

I think that now we immediately need to use super-graph as an importable library in any Go project (maybe as a simple http endpoint) because we would also increase the number of people who use it and therefore its testability on the battlefield.

After this it would be nice to open another thread on plugins and other long-term maintainability techniques.

[dosco]

Great news to share, Super Graph is now a GO library that you can include in your own code. Fetch data in your own code with GraphQL instead of struggling with ORMs or complex SQL.

Checkout this example:

package main

import (
    "database/sql"
    "fmt"
    "time"
    "github.com/dosco/super-graph/core"
    _ "github.com/jackc/pgx/v4/stdlib"
)

func main() {
    db, err := sql.Open("pgx", "postgres://postgrs:@localhost:5432/example_db")
    if err != nil {
        log.Fatalf(err)
    }

    conf, err := core.ReadInConfig("./config/dev.yml")
    if err != nil {
        log.Fatalf(err)
    }

    sg, err = core.NewSuperGraph(conf, db)
    if err != nil {
        log.Fatalf(err)
    }

    query := `
        query {
            posts {
            id
            title
        }
    }`

    res, err := sg.GraphQL(context.Background(), query, nil)
    if err != nil {
        log.Fatalf(err)
    }

    fmt.Println(string(res.Data))
}

[frederikhors]

OMG!

[frederikhors]

I'm testing it... I'll let you know ASAP!

Amazing work! Amazing work!

[frederikhors]

I tried very quickly to integrate it into a project I'm working on (I'll continue later).

It would be great to be able to make it become a middleware (https://www.alexedwards.net/blog/making-and-using-middleware) in a middleware chain of any Go HTTP server.

Example

Now my server has this structure:

main () {
  setup()

  router := newRouter() // Gin, Echo, Chi, whatever

  router.Use(...middlewares...)

  // endpoints for GraphQL
  router.Group(... {
    router.Post(... handler for POST GraphQL calls)
  })  
}

How do you think I can use SuperGraph in this scenario?

Considering also that the endpoint for users must always be only one (e.g. localhost / graphql).

Perhaps a solution may be to analyze the incoming query and see if it is among "SuperGraph's", then use middleware before the line:

router.Post(... handler for POST GraphQL calls)

or let it go to the next middleware that can handle it.

What do you think about it?

Did you have other use cases in mind that I didn't think of?

[frederikhors]

Your example here: https://supergraph.dev/guide.html#stripe-api-example

having SuperGraph inside my project is a game changer!

I can call my own stripe() client internally, not using HTTP calls!

Am I wrong?

[dosco]

It should be relatively easy to build a middleware depending on the GOLang HTTP framework you are using. Below is an example I found if you're using standard lib. Thanks or the feedback I'm equally excited launching this and using it myself. You don't have to make it a middleware just use it in the handlers for you're existing REST endpoints in place of whatever ORM you might have used.

func supergraphMiddleware(next http.Handler) http.Handler {
  <initial super graph here>

  return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    <call the GraphQL function here>

    next.ServeHTTP(w, r)
  })
}

[dosco]

I can call my own stripe() client internally, not using HTTP calls!

Yes you should be able to,

[howesteve]

That is great. Is there a way to use core.NewSuperGraph() directly with a *pgxpool.Pool instance instead of *sql.DB? One would use pgx's stdlib.AcquireConn(), but on PGX4 they've made pgxpool.Pool the default pool and stidlib.AcquireConn() works only with *pgx.Pool and not *pgxpool.Pool.

Thanks.
Howe

[dosco]

@howesteve funny story originally the Super Graph service used *pgxpool.Pool internally everywhere. I changed it to the more generic *sql.DB when I extracted the core API out. So now I use pgx/stdlib. Honestly their docs are very unclear on what the difference is as far as I know stdlib has a built in pool.

Maybe you could write your own sql.DB driver that wraps pgx.Pool?

[howesteve]

Sure, I'll work on it. But I was thinking that when (and if) subscriptions support arrive, LISTEN/NOTIFY support (i.e. some kind of conn.Listen()) method will be needed anyway, and that is not supported by sql.DB...

[dosco]

I'm working on some early design ideas for subscriptions focus is on making it scale. Early experiments tell me that listen notify is does not scale as well and polling seems be the way to go here.

[howesteve]

Nice. Actually that seems to be the same conclusion of the hasura team when then implemented it two years ago:

https://github.com/hasura/graphql-engine/blob/master/architecture/live-queries.md

But I think listen/notify could scale better on some heavier loads. Ex:
50000 clients checking if a message arrived for their own user. That will require comparing a database field to a session variable, so one query will have to be issued for each client each time. If the polling timer is 500ms, that is 100000 queries per second? On heavier loads, that doesn't seem to end up well. It would be super expensive on servers that charge per database read.

Postgraphile guys chose a plugin-based approach and support LISTEN/NOTIFY or WAL monitoring:
https://www.graphile.org/postgraphile/live-queries/

Each approach comes with its pros and cons; polling always give perfect results but could easily overload the server, WAL requires extra client side programming and can only inspect physical columns, and LISTEN/NOTIFY require extra setup steps in the connection and configuring triggers.

Not a clear win here...

[dosco]

Thanks for those links I'll take a deeper look. Having read up on some of this stuff it seems if you poll using a join query where the original subscribed query gets its variables (user id, etc) from the joined table then you are not executing 100k queries but just 1 with 100k rows one per client. In theory this sounds more efficient to me on small and large subscription counts.

[lostb1t]

while obvious off-topic I think polling is a very valid first approach and quick to implement.
Other implementations can always be added later on.

[frederikhors]

I think we can close this, @dosco.

[dosco]

I was keeping it open just till we land the public API for the service too,

[dosco]

Closing this since the Serv package is also a library.