CLI options organization

[labra]

I would like to raise some discussion about the organization of the CLI design structure.

Design goals

Some design rationale about the CLI options:

• rudof should have an intuitive interface, with a set of options that is easy to remember and stable.
• It should be possible to extend the options with new features that appear in the future
• It should provide a set of good defaults that work well in general but where most of the things can be customized
• rudof has commands which have their own set of parameters. Most of the commands can be classified in two groups:
  • verbs that ask rudof to do some action, for example: query, validate, etc.
  • names that get information about some technology, for example: shex, shacl, etc.

General structure and scope

The main scope of rudof is to provide tools to help work with graph data technologies to improve their quality by enabling to check the conformance of data to some data models, schemas or shapes.

We are considering the following possibilities:

Verbs

• query: Implement queries from Knowledge Graphs. We currently support SPARQL by default, but we could add support for other query languages in the future like OpenCypher
• validate: Validate a Knowledge graph with some schema or set of shapes. We support some specific variants like shex-validate, shacl-validate, pgschema-validate which contain better defaults for those technologies
• generate: Generate a Knowledge Graph from some input.
• convert: Convert some data model or schema to a different one
• compare: Compare some data model or schema with another one

More commands that we are planning to add in the future:

• extract: Extract/infer data models from data. This command will be based on ShEx.

Commands that are names of specific data models or schema technologies

We define commands for some specific data models or technologies which could be used for linting or validating those data models without a concrete knowledge graph:

• shex: Manage ShEx schemas
• shacl: Manage SHACL shapes graphs
• dctap: Manage DCTAP models
• rdf-config: Manage rdf-config schemas
• pgschema: Manage property graph schemas

In the future we would like to include support to at least:

• linkml: Manage LinkML definitions

Commands that are names that are exceptions

The following commands are names of technologies but those technologies are not data models or shapes.

• shapemap: Manage shape maps which are used in ShEx

Commands that are generic names (not verbs)

• data: Get information about graph data like an RDF files, or property graphs. This command can have a parameter to specify the type of technology used (by default, RDF).
• node: Get information about a node in a graph. This command is very useful for obtaining information about RDF nodes.

Exceptions

There are some exceptions to the above:

• mcp: This command exports rudof functionality as an MCP server
• service: This command gets information from a SPARQL endpoint service.
• materialize: This command can generate an RDF graph from a ShEx schema and semantic-action data to support ShExMap extension

In the future, we are planning to add:

• server: Export rudof functionality as a Web service

After writing the overview, I realize that there are already some exceptions and redundancies that we could probably avoid:

• The materialize option could probably be merged with the generate
• The data option can be very generic if we start adding support for different implementations of property graphs.

Other ideas?

[adsharma]

Flow I'm thinking about for LPG graphs:

1. validate: rudof validate examples/user.ttl --schema examples/user.shex --shapemap examples/user.sm
2. connect: rudof connect backend dbname --type ... <url>
3. cypher-ddl: rudof cypher-ddl --schema examples/user.shex --shapemap examples/user.sm
4. gql-ddl: rudof gql-ddl --schema examples/user.shex --shapemap examples/user.sm
5. load: rudof load examples/user.ttl

Subsequent mutations on the graph in the LPG database will be constrained by the generated DDL. This is generally necessary, but not fully sufficient to ensure validation.

6. validate-cypher-db: rudof validate dbname --schema examples/user.shex --shapemap examples/user.sm

[labra]

Flow I'm thinking about for LPG graphs:

Thanks for the workflow, I will try to check which of those options we already support or how they could be supported.

1. validate: rudof validate examples/user.ttl --schema examples/user.shex --shapemap examples/user.sm

As far as I know this one is already supported by rudof.

2. connect: rudof connect backend dbname --type ... <url>

This one is vrey interesting and raises another issue that we are not yet sure how to tackle, which is the way we handle state in the rudof CLI. At this moment, the rudof CLI is trying to be stateless with the goal of being deterministic. So we don't offer a connect command that generates a connection that can later be reused. I have just created another discussion thread for this possibility and I would be happy to know your opinion.

3. cypher-ddl: rudof cypher-ddl --schema examples/user.shex --shapemap examples/user.sm
4. gql-ddl: rudof gql-ddl --schema examples/user.shex --shapemap examples/user.sm

As I understand the goal here is to have a ddl command specialized for different types of query languages (cypher, gql, etc.) which creates a set of constraints or shapes for the database. I would probably prefer something like

• rudof ddl --technology cypher --schema user.shex --shapemap user-sm
• rudof ddl --technology gql --schema user.shex --shapemap user-sm

Which would allow other technologies in the future...

5. load: rudof load examples/user.ttl

Subsequent mutations on the graph in the LPG database will be constrained by the generated DDL. This is generally necessary, but not fully sufficient to ensure validation.

If I understand, your goal here is that once the DDL has been defined and the connection established, future attempts to load data would validate the data before it is inserted in the database. Is that what you would like to accomplish with the load command?

6. validate-cypher-db: rudof validate dbname --schema examples/user.shex --shapemap examples/user.sm

This one would probably be handled in the same way as the generic validate but reusing the connection details to the database and maybe reusing the DDL specified.

[adsharma]

1. Yes - existing
2. You could persist the connection url to a file, so subsequent commands can access
3. (and 4) Making it generic make sense. But technology could be anything like quantum
4. Covered by 3
5. load is a generally a copy command. The DDL that the db already has will throw if something doesn't conform.
6. Yes - that's the idea. Another scan of the database.

Some people prefer the use of triggers and stored procedures to enforce constraints DDL can't express. But I'm not sure graph databases have the sufficient level of expressivity to do so today (vs SQL)

[adsharma]

There are also efforts like LEX that seek to bring these SHACL validation features to LPG based on the belief that gql-ddl is insufficient.

https://www.youtube.com/watch?v=GG3JvPiFudQ&t=139s

Personally I don't have a lot of time to engage in the LPG only validation efforts because of competing priorities. But if there are low hanging fruit that bridge the obvious gap between storage/retrieval oriented graph technologies (Cypher, GQL) and semantic graph technologies (RDF/SHACL), I'd like to invest some time. It makes sense to layer semantics on more advanced graph storage tech instead of inventing new RDF storage (6 indexes on various permutation of triples).

[labra]

There are also efforts like LEX that seek to bring these SHACL validation features to LPG based on the belief that gql-ddl is insufficient.

https://www.youtube.com/watch?v=GG3JvPiFudQ&t=139s

Personally I don't have a lot of time to engage in the LPG only validation efforts because of competing priorities. But if there are low hanging fruit that bridge the obvious gap between storage/retrieval oriented graph technologies (Cypher, GQL) and semantic graph technologies (RDF/SHACL), I'd like to invest some time. It makes sense to layer semantics on more advanced graph storage tech instead of inventing new RDF storage (6 indexes on various permutation of triples).

Thanks for the pointer to LEX. It looks similar to the route that we have already taken in rudof to add support to PGSchema and to extend PGSchema with property constraints. You can see a paper introducing the concept here and a short paper describing a demo we gave at that conference.

Currently rudof already supports property graph data which follows the YARS-PG syntax and validation with PGSchema schemas. This feature is described here.

[adsharma]

#779 implements some of the ideas in this discussion