Implement RDMS backend (postgres/mysql/sqlite)

[ivanvanderbyl]

Description of the PR

(WIP) Implements a backend using Ent that supports common RDMS such as Postgres, MySQL, Sqlite. Tested only with Postgres at the moment.

PR Checklist

• All commits have a Developer Certificate of Origin (DCO) -- they are generated using -s flag to git commit.
• All new changes are covered by tests
• If GraphQL schema is changed, make generate has been run
• If collectsub protobuf has been changed, make proto has been run
• All CI checks are passing (tests and formatting)
• All dependent PRs have already been merged

To implement:

• Migration script (Atlas or ent)
• Test harness using txdb
• Artifacts
• Builders
• Cve
• Ghsa
• Osv
• Packages
• Sources
• CertifyBad
• CertifyGood
• CertifyVEXStatement
• CertifyVuln
• HasSBOM
• HasSlsa
• HasSourceAt
• HashEqual
• IsDependency
• IsOccurrence
• IsVulnerability
• PkgEqual
• Scorecards
• IngestArtifact
• IngestArtifacts
• IngestBuilder
• IngestCve
• IngestGhsa
• IngestMaterials
• IngestOsv
• IngestPackage
• IngestPackages
• IngestSource
• CertifyScorecard
• IngestCertifyBad
• IngestCertifyGood
• IngestDependency
• IngestDependencies
• IngestHasSbom
• IngestHasSourceAt
• IngestHashEqual
• IngestIsVulnerability
• IngestOccurrence
• IngestOccurrences
• IngestPkgEqual
• IngestSLSA
• IngestVEXStatement
• IngestVulnerability
• Neighbors
• Node
• Nodes
• Path
• FindSoftware
• Implement new model.MatchType on deps
• Implement new vulnerability changes

Note to reviewers: A large chunk of this is code generated using Ent, which results in a large change surface. You'll notice that most ent code is identical for each node type, so you can skim it easily. The actual Backend implementation is in pkg/assembler/backends/ent/backend with the generated code in the parent directory.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[pxp928]

Thanks, @ivanvanderbyl! We never knew about this. This could be a game changer for backend resolver implementation work.

[lumjjb]

Nice!!!! This is super awesome!!! Looking forward to diving deeper into this! Let us know if you'd like to spend time to run through anything that you have questions on!

[mihaimaruseac]

This is amazing! Thank you for starting this!

[ivanvanderbyl]

@lumjjb one area that isn't so clear is the schema, I'm inferring a lot of this from reading the inmem and neo4j implementations. Do you have a design document for this?

Another area to discuss is how Ent can optimally work with gqlgen. Given the current Backend interface, if we make a query like (take from the docs):

packages(pkgSpec: { type: "deb" }) {
    type
    namespaces {
      namespace
    }
  }

The backing Ent query should look like:

records,err := client.Package.Query().
  Where(package.TypeEQ(typ)).
  WithNamespaces().
  All(ctx)

Key to this is knowing that the user asked for namespaces and only eagerly querying that edge when asked. The current Backend API I believe makes this possible? Ensuring we don't create an N+1 query problem, which is quite easy to do with gqlgen 🦶🏻 🔫

A larger design decision would be to have Ent generate all this glue code between Ent and gqlgen if you don't mind the interface coupling (https://entgo.io/docs/tutorial-todo-gql-field-collection)

[lumjjb]

Key to this is knowing that the user asked for namespaces and only eagerly querying that edge when asked. The current Backend API I believe makes this possible?

Yea if I understand it right. The software trees like packages, etc. all have this behavior.

A larger design decision would be to have Ent generate all this glue code between Ent and gqlgen if you don't mind the interface coupling (https://entgo.io/docs/tutorial-todo-gql-field-collection)

I do personally like being able to manually have control over this since, I do envision that some graphql APIs could be tricky to optimize with ent generated code. With current APIs it looks like a nice fit, but I think we would add some more "analytics" type queries which may make things a bit tricky.

Do you have a design document for this?

hmm, have you looked at some of these docs by any chance? If it still isn't clear maybe we can have a call and answer some questions and then add to the docs to make things clearer!

• GraphQL Doc
• Ontology doc
• Ontology Definitions

[pxp928]

Yea if I understand it right. The software trees like packages, etc. all have this behavior.

+1 Via the graphQL preloads, we can get the query fields that the user requested and optimize based on that. We did something similar in the neo4j implementation.

[mihaimaruseac]

Key to this is knowing that the user asked for namespaces and only eagerly querying that edge when asked. The current Backend API I believe makes this possible?

Yes, we have a getPreloads function for this, but this can be adapted if needed

[ivanvanderbyl]

👍🏻 for preloads.

Thanks for pointing me to the docs, hadn't dug deep enough to find them yet. Big help! I've added the software tree packages and a tests. I've also configured this to do an upsert so that you can call ingest multiple times without duplicating each tree node.

I think this is starting to take shape.

[mrizzi]

@pxp928 makes sense. How are you handling the database services for testing arango/neo etc? Is it just a matter of adding a postgres service to Github actions?

Maybe it's out of scope for this PR but I would suggest to evaluate testcontainers.
It would help a lot without having to rely on GH actions but relying on a solution that would easily work also locally for developers.

[pxp928]

Maybe it's out of scope for this PR but I would suggest to evaluate testcontainers.
It would help a lot without having to rely on GH actions but relying on a solution that would easily work also locally for developers.

Interesting. I like the idea!

[mrizzi]

So if @ivanvanderbyl is more confident with GH Services, for this PR, it's fine to add them in order to have the Postresql instance to run tests with.

Then, later on, I can work on adding Testcontainers support to guac tests for Ent and maybe also for ArangoDB to have consistency across development and testing with different backends.

[jeffmendoza]

Lets merge this into main and continue there for the finishing touches.