Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TypeDB 3.0 Roadmap #6764

Open
flyingsilverfin opened this issue Mar 16, 2023 · 10 comments
Open

TypeDB 3.0 Roadmap #6764

flyingsilverfin opened this issue Mar 16, 2023 · 10 comments

Comments

@flyingsilverfin
Copy link
Member

flyingsilverfin commented Mar 16, 2023

Problem to Solve

We collect the agreed list of changes and requirements that will be in the first version of TypeDB 3.0.

Changes

API

Driver

TypeQL

Value restriction:

Require further discussion:

Relation implementation

Changes proposed and rejected:

@haikalpribadi
Copy link
Member

Let's make sure each of them is documented properly in an issue, @flyingsilverfin

@flyingsilverfin
Copy link
Member Author

Yes @haikalpribadi that's what the colons are for :D I have to get to that next

@flyingsilverfin
Copy link
Member Author

flyingsilverfin commented Mar 17, 2023

Internal changes

A ? indicates not yet fully discussed.

TypeQL

  • Fix backslash escaping
    ? - [ ] Allow modifiers on match inside of a delete/insert query to allow flexible query operations such as batching
    ? - [ ] Rename MatchQuery to GetQuery. To discuss: how this plays with the above

RPC

  • Replace the session ID with a long, instead of an inefficient vector

Pattern & Resolvables

  • Implement the query representation as a set of constraints that own variables, instead of the other way around
  • Remove the idea of Resolvables (eg. Concludables, etc.) and merge them into Patterns

** Concepts **

  • The schema concept layer should more aggressively cache, in more CPU friendly format, various shortcuts such as as owned attribute types directly, without having to traverse through the super types as well. Additionally, we should store all schema-level data in flat sorted arrays (likely never exceeds about 100mb in size with the largest possible schemas) to optimise access.

Traversal & Reasoner

  • Rearchitect reasoner to manage its own memory
  • Push Concept to be the bottom layer of the database that traversals and reasoner operate over. Below that can still exist a graph and storage layer, but they should not be exposed
  • Convert explain into a explain() query that takes a query and bounds. Alternatively, we could just explain the existence of an inferred concept without a query? Also, convert explanations into something more native?
  • Handle negations in the traversal natively

@lveillard
Copy link

lveillard commented Mar 17, 2023

Thank you guys for working on this and sharing it with us.
The two key features that extremely limit our use cases and I'm missing in this list:

Optionals/fetch

#6322
Including a way to have optional played roles also, not only attributes.

Vectors and ordered lists

I see they've been discarded :S, but there is no simple workaround for this:
#6327

Vectors

We don't need them as a particular attribute types, maybe a @sortable or @indexable when defining relations would work.

Storing something like this in typeDB is really hard, and mutate it (add items at particular positions) in a performant way is near to impossible
image

Ordered lists

This also includes ordered lists with repeated values which are really hard to store in typeDB.
Ex: [1,2,2,3,7,2] or ['blue', 'green', 'green', 'red']

@brettforbes
Copy link

brettforbes commented Nov 21, 2023

The idea of banning attributes from playing roles is a disaster idea, forcing us back into the nightmare modelling world were everything must be an entity. Please god do not remove role-playing from attributes!!!!!! There is no benefit, and so much downside.

I can show you plenty of examples where this feature is essential, that cannot be solved by structs!!!!! Why become just like the dumbo graphs and force us into false modelling ideas, where everything must be an entity, even if it is not a thing??? It is easy to derive a false premise when examining toy problems. This goes against Type Theory as propounded by Christoph!!

How are you better than Neo4j if you make this move??? This is the single-most distressing conceptual idea I have ever seen. Terrible idea

@brettforbes
Copy link

brettforbes commented Nov 23, 2023

An example of why attributes that play roles are so powerful. this could not be done in any other database. Its a part of a schema where a language application links a dictionary meaning (sense) to all of its relatives. Just look at thow powerful typeDB is.

So if you take away this ability, how will you solve this schema??? @flyingsilverfin Do we need to look for a new database that can handle this?

define
#############################################
# define word-sense network
    grammar sub attribute, value string;
    description sub attribute, value string;
    sense sub attribute, value string,
        owns grammar,
        owns description,
        plays Means:interpretant,
        plays Attributes:owner,
        plays Hypernyms:Specific,
        plays Hypernyms:General,
        plays Hyponym:General,
        plays Hyponym:Specific,
        plays Holonyms:Part,
        plays Holonyms:Whole,
        plays Meronyms:Whole,
        plays Meronyms:Part,
        plays SubstanceHolonym:Part,
        plays SubstanceHolonym:Whole,
        plays SubstanceMeronyms:Whole,
        plays SubstanceMeronyms:Part,
        plays Entails:This,
        plays Entails:That,
        plays EntailedBy:That,
        plays EntailedBy:This,
        plays Causing:This,
        plays Causing:That,
        plays Resulting:That,
        plays Resulting:This,
        plays Causes:This,
        plays Causes:That,
        plays CausedBy:That,
        plays CausedBy:This,
        plays NounOfNoun:Super,
        plays NounOfNoun:Subordinate,
        plays NounHaveNoun:Subordinate,
        plays NounHaveNoun:Super,
        plays TableUndergoerAdj:Adjective,
        plays TableUndergoerAdj:Undergoer,
        plays TableAdjUndergoer:Undergoer,
        plays TableAdjUndergoer:Adjective,
        plays TableActorVerb:Actor,
        plays TableActorVerb:Verb,
        plays TableVerbActor:Verb,
        plays TableVerbActor:Actor,
        plays TableVerbUndergoer:Undergoer,
        plays TableVerbUndergoer:Verb,
        plays TableUndergoerVerb:Verb,
        plays TableUndergoerVerb:Undergoer,
        plays TableVerbAdjective:Verb,
        plays TableVerbAdjective:Adjective,
        plays TableAdjectiveVerb:Adjective,
        plays TableAdjectiveVerb:Verb,
        plays TableVerbUse:Verb,
        plays TableVerbUse:Use,
        plays TableUseVerb:Use,
        plays TableUseVerb:Verb;
        
    word sub attribute, value string,
        plays Means:sign,
        plays Attributes:owner;```

@brettforbes
Copy link

What is the reason for removing the ability for multiple role owners of the same type. Once again, you break all of our code, but this time there is no alternative for a list of relations. Why do you keep breaking our code base. what possible benefit is there? Do you have an alternative, because every one of our 90 database objects uses this feature.

This one is terrible because it breaks a lot of use cases, and it inhibits the use of list of relations. There seems to be no alternative for a list of relations, so once again there is breaking code with a terrible alternative, instead of one relation with a list of roles, i need multiple relations.

Consider a simple scenario, where i have lists of references that get turned into relations, so

{
      "type": "incident",
      "spec_version": "2.1",
      "id": "incident--8e94bc9d-79d6-4612-aa97-1fab55f35d76",
      "created": "2023-11-18T05:26:43.46768Z",
      "modified": "2023-11-18T05:26:43.46768Z",
      "name": "potential phishing",
      "extensions": {
            "extension-definition--ef765651-680c-498d-9894-99799f2fa126": {
                  "extension_type": "property-extension",
                  "investigation_status": "new",
                  "determination": "suspected",
                  "incident_types": [
                        "dissemination-phishing-emails"
                  ],
                  "sequence_start_refs": [
                        "sequence--58babacb-4e15-480e-a73c-3eac9f6e2d52"
                  ],
                  "sequence_refs": [
                        "sequence--58babacb-4e15-480e-a73c-3eac9f6e2d52",
                        "sequence--90ceef43-4484-4126-af05-9c9d60290a36",
                        "sequence--d41c2e05-da07-42ce-85bb-05fa280f3f9f",
                        "sequence--e84228de-4831-4f98-9f28-1a51744ec4b9",
                        "sequence--9fac1cf7-db8b-4f74-bbd0-14a036459cc3",
                        "sequence--edde80dd-c68f-47c9-bd3c-39ed461700a6",
                        "sequence--5d59d1e4-1bce-4599-a763-f05fefa6a9f1",
                        "sequence--1521a7d0-c597-4a8a-a074-6d33337b627e",
                        "sequence--cf28bb62-ebba-4a6e-b0d1-c25786f23e92",
                        "sequence--e5336bdf-238c-4ff2-9db7-9f7ca90348e7"
                  ],
                  "event_refs": [
                        "event--236d689a-aaa9-4bc1-8401-d86cb2d3c430"
                  ],
                  "other_object_refs": [
                        "email-addr--64157c44-5eb7-5d01-8710-b288f8ccb625",
                        "user-account--41b2b7c4-6950-5d22-88e0-b376c93097d0",
                        "email-addr--df33039b-3a3c-5653-9cd1-13923a63fcba",
                        "email-message--5cb8312b-26e9-5346-8794-936f3f5288c8",
                        "url--3279c7de-8f91-5c1a-99d9-d6546c6c41f7",
                        "relationship--a8a2a45a-e3a9-4ebc-a1f5-4fdf91960fd4",
                        "observed-data--c76d0cae-3c1c-4593-9aa1-8963b63e4776",
                        "indicator--3f624ff1-7c2e-40db-8d7c-acc8723afe95",
                        "sighting--49cc5c32-98b8-4d15-94bf-2f867d71a2fa",
                        "identity--b30fc24d-4d5a-49a4-8aee-155788edbae3",
                        "anecdote--e1298bc0-818e-5cdb-9154-eac37c8e260f",
                        "observed-data--fb077ab9-4a21-4a3d-91b1-02bfb9c6c493",
                        "sighting--d0a71341-cf9b-474f-8cdb-eba6d9e96a68",
                        "user-account--47df6d33-0005-5b45-9875-e7a609b12e9b",
                        "user-account--47df6d33-0005-5b45-9875-e7a609b12e9b",
                        "user-account--74f5352e-ea3f-526c-8430-1cbb9ccd3109",
                        "email-addr--b8048a91-def8-5ef7-8cd3-a7b3db9278e5",
                        "user-account--5aaaa4e2-0974-5ab4-9069-41a16197f0ff",
                        "email-addr--30d9a416-203b-55c8-b796-8eb65ab5275e",
                        "sighting--575ca2c4-7d0f-4984-86a1-e0f7422f6237",
                        "relationship--9789896d-1c02-485f-88e3-7bc6c749f284",
                        "relationship--9789896d-1c02-485f-88e3-7bc6c749f284",
                        "relationship--524cee6e-3108-4c27-97f3-d0f290f4324c",
                        "observed-data--7a59a0f6-448b-45c6-ac4b-2c6ce71d439f",
                        "identity--05bf6d14-9483-423d-b4f3-b19f420c0803",
                        "ipv4-addr--a40768a0-5841-50d2-838d-bcb35ee92131",
                        "domain-name--aa90a5c6-6a78-520b-bcfd-e2bf0c25bce1",
                        "observed-data--232ef32d-3b74-4404-92e9-a04a6da2edb6",
                        "indicator--00c82e87-a7c4-4cbb-b58a-577d988dd84e",
                        "sighting--feb6fbd9-ff3f-45f8-ae59-2976fc766196",
                        "location--7cd4b991-d086-47a0-a430-c92f17a0c2e1",
                        "identity--da2ce49d-c608-4126-8efe-dbf96b5f070f",
                        "relationship--c5bcfa3f-2086-4122-a1f1-dc46ace79baa",
                        "sighting--791d2177-e8ae-4021-ac17-5a68f5ccabed",
                        "software--3974613c-1980-5209-8f08-17920a04da04",
                        "file--e7c7ddeb-d693-53e7-bcd8-f40951888120",
                        "process--fe3c9253-8490-4647-9b45-41a918fb6e15",
                        "relationship--2ca1e438-492d-45a1-b0a9-7285db39342f",
                        "relationship--1995ce47-c1bf-401b-944f-9661c9da43a2",
                        "relationship--50929fe5-29e9-4433-9a74-ae834e96f00c",
                        "observed-data--c177cd97-7226-4572-a60a-5caf70f49873",
                        "sighting--7270fd2c-05c7-401d-a553-deea53d5ecda"
                  ],
                  "task_refs": [
                        "task--3d00a36b-506f-4ccb-8888-ec8d099f399d",
                        "task--0b663052-5d31-4c42-9f06-7343985cce6c",
                        "task--44a9d7fa-028d-4fa0-b822-32680eda7d7e",
                        "task--43b3b336-4774-4267-9ffd-175f83b7cf7c",
                        "task--6be4f7f4-8ece-4b94-9ec7-e7009e38c547",
                        "task--6be4f7f4-8ece-4b94-9ec7-e7009e38c547"
                  ],
                  "impact_refs": [
                        "impact--86625ce2-4b1f-4fc7-8a6b-fc70e086c006",
                        "impact--d82add0b-eb43-42d3-98c0-ae73ca232bf8"
                  ]
            }
      }
}```

Now the typeql for this looks like this.

match
$task00 isa task, has stix-id "task--3d00a36b-506f-4ccb-8888-ec8d099f399d";
$task10 isa task, has stix-id "task--0b663052-5d31-4c42-9f06-7343985cce6c";
$task20 isa task, has stix-id "task--44a9d7fa-028d-4fa0-b822-32680eda7d7e";
$task30 isa task, has stix-id "task--43b3b336-4774-4267-9ffd-175f83b7cf7c";
$task40 isa task, has stix-id "task--6be4f7f4-8ece-4b94-9ec7-e7009e38c547";
$task50 isa task, has stix-id "task--6be4f7f4-8ece-4b94-9ec7-e7009e38c547";
$event00 isa event, has stix-id "event--236d689a-aaa9-4bc1-8401-d86cb2d3c430";
$impact00 isa impact, has stix-id "impact--86625ce2-4b1f-4fc7-8a6b-fc70e086c006";
$impact10 isa impact, has stix-id "impact--d82add0b-eb43-42d3-98c0-ae73ca232bf8";
$email-addr00 isa email-addr, has stix-id "email-addr--64157c44-5eb7-5d01-8710-b288f8ccb625";
$user-account10 isa user-account, has stix-id "user-account--41b2b7c4-6950-5d22-88e0-b376c93097d0";
$email-addr20 isa email-addr, has stix-id "email-addr--df33039b-3a3c-5653-9cd1-13923a63fcba";
$email-message30 isa email-message, has stix-id "email-message--5cb8312b-26e9-5346-8794-936f3f5288c8";
$url40 isa url, has stix-id "url--3279c7de-8f91-5c1a-99d9-d6546c6c41f7";
$stix-core-relationship50 isa stix-core-relationship, has stix-id "relationship--a8a2a45a-e3a9-4ebc-a1f5-4fdf91960fd4";
$observed-data60 isa observed-data, has stix-id "observed-data--c76d0cae-3c1c-4593-9aa1-8963b63e4776";
$indicator70 isa indicator, has stix-id "indicator--3f624ff1-7c2e-40db-8d7c-acc8723afe95";
$sighting80 isa sighting, has stix-id "sighting--49cc5c32-98b8-4d15-94bf-2f867d71a2fa";
$identity90 isa identity, has stix-id "identity--b30fc24d-4d5a-49a4-8aee-155788edbae3";
$anecdote100 isa anecdote, has stix-id "anecdote--e1298bc0-818e-5cdb-9154-eac37c8e260f";
$observed-data110 isa observed-data, has stix-id "observed-data--fb077ab9-4a21-4a3d-91b1-02bfb9c6c493";
$sighting120 isa sighting, has stix-id "sighting--d0a71341-cf9b-474f-8cdb-eba6d9e96a68";
$user-account130 isa user-account, has stix-id "user-account--47df6d33-0005-5b45-9875-e7a609b12e9b";
$user-account140 isa user-account, has stix-id "user-account--47df6d33-0005-5b45-9875-e7a609b12e9b";
$user-account150 isa user-account, has stix-id "user-account--74f5352e-ea3f-526c-8430-1cbb9ccd3109";
$email-addr160 isa email-addr, has stix-id "email-addr--b8048a91-def8-5ef7-8cd3-a7b3db9278e5";
$user-account170 isa user-account, has stix-id "user-account--5aaaa4e2-0974-5ab4-9069-41a16197f0ff";
$email-addr180 isa email-addr, has stix-id "email-addr--30d9a416-203b-55c8-b796-8eb65ab5275e";
$sighting190 isa sighting, has stix-id "sighting--575ca2c4-7d0f-4984-86a1-e0f7422f6237";
$stix-core-relationship200 isa stix-core-relationship, has stix-id "relationship--9789896d-1c02-485f-88e3-7bc6c749f284";
$stix-core-relationship210 isa stix-core-relationship, has stix-id "relationship--9789896d-1c02-485f-88e3-7bc6c749f284";
$stix-core-relationship220 isa stix-core-relationship, has stix-id "relationship--524cee6e-3108-4c27-97f3-d0f290f4324c";
$observed-data230 isa observed-data, has stix-id "observed-data--7a59a0f6-448b-45c6-ac4b-2c6ce71d439f";
$identity240 isa identity, has stix-id "identity--05bf6d14-9483-423d-b4f3-b19f420c0803";
$ipv4-addr250 isa ipv4-addr, has stix-id "ipv4-addr--a40768a0-5841-50d2-838d-bcb35ee92131";
$domain-name260 isa domain-name, has stix-id "domain-name--aa90a5c6-6a78-520b-bcfd-e2bf0c25bce1";
$observed-data270 isa observed-data, has stix-id "observed-data--232ef32d-3b74-4404-92e9-a04a6da2edb6";
$indicator280 isa indicator, has stix-id "indicator--00c82e87-a7c4-4cbb-b58a-577d988dd84e";
$sighting290 isa sighting, has stix-id "sighting--feb6fbd9-ff3f-45f8-ae59-2976fc766196";
$location300 isa location, has stix-id "location--7cd4b991-d086-47a0-a430-c92f17a0c2e1";
$identity310 isa identity, has stix-id "identity--da2ce49d-c608-4126-8efe-dbf96b5f070f";
$stix-core-relationship320 isa stix-core-relationship, has stix-id "relationship--c5bcfa3f-2086-4122-a1f1-dc46ace79baa";
$sighting330 isa sighting, has stix-id "sighting--791d2177-e8ae-4021-ac17-5a68f5ccabed";
$software340 isa software, has stix-id "software--3974613c-1980-5209-8f08-17920a04da04";
$file350 isa file, has stix-id "file--e7c7ddeb-d693-53e7-bcd8-f40951888120";
$process360 isa process, has stix-id "process--fe3c9253-8490-4647-9b45-41a918fb6e15";
$stix-core-relationship370 isa stix-core-relationship, has stix-id "relationship--2ca1e438-492d-45a1-b0a9-7285db39342f";
$stix-core-relationship380 isa stix-core-relationship, has stix-id "relationship--1995ce47-c1bf-401b-944f-9661c9da43a2";
$stix-core-relationship390 isa stix-core-relationship, has stix-id "relationship--50929fe5-29e9-4433-9a74-ae834e96f00c";
$observed-data400 isa observed-data, has stix-id "observed-data--c177cd97-7226-4572-a60a-5caf70f49873";
$sighting410 isa sighting, has stix-id "sighting--7270fd2c-05c7-401d-a553-deea53d5ecda";

insert $incident isa incident,
has stix-type $stix-type,
has spec-version $spec-version,
has stix-id $stix-id,
has created $created,
has modified $modified,
has name $name;

$stix-type "incident";
$spec-version "2.1";
$stix-id "incident--8e94bc9d-79d6-4612-aa97-1fab55f35d76";
$created 2023-11-18T05:26:43.467;
$modified 2023-11-18T05:26:43.467;
$name "potential phishing";

$incident-ext isa incident-ext,
has extension-type $extension-type,
has investigation-status $investigation-status,
has determination $determination,
has incident-types $incident_types0;

$extension-type "property-extension";
$investigation-status "new";
$determination "suspected";
$incident_types0 "dissemination-phishing-emails";

$occurence-ext0 (occurence:$incident, incident-core:$incident-ext) isa occurence-ext;

$task-refs0 (incident:$incident-ext, the-task:$task00, the-task:$task10, the-task:$task20, the-task:$task30, the-task:$task40, the-task:$task50) isa task-refs;

$event-refs0 (incident:$incident-ext, the-event:$event00) isa event-refs;

$impact-refs0 (incident:$incident-ext, the-impact:$impact00, the-impact:$impact10) isa impact-refs;

$other-obj-refs0 (object:$incident-ext, referred:$email-addr00, referred:$user-account10, referred:$email-addr20, referred:$email-message30, referred:$url40, referred:$stix-core-relationship50, referred:$observed-data60, referred:$indicator70, referred:$sighting80, referred:$identity90, referred:$anecdote100, referred:$observed-data110, referred:$sighting120, referred:$user-account130, referred:$user-account140, referred:$user-account150, referred:$email-addr160, referred:$user-account170, referred:$email-addr180, referred:$sighting190, referred:$stix-core-relationship200, referred:$stix-core-relationship210, referred:$stix-core-relationship220, referred:$observed-data230, referred:$identity240, referred:$ipv4-addr250, referred:$domain-name260, referred:$observed-data270, referred:$indicator280, referred:$sighting290, referred:$location300, referred:$identity310, referred:$stix-core-relationship320, referred:$sighting330, referred:$software340, referred:$file350, referred:$process360, referred:$stix-core-relationship370, referred:$stix-core-relationship380, referred:$stix-core-relationship390, referred:$observed-data400, referred:$sighting410) isa other-obj-refs;


As you can see, everyone of those id lists turns into a relation with repeated roles. Why are you going to break yet another feature in your database?? What is the alternative for a relation that is a list? Repeated relations is not the same

@lveillard
Copy link

lveillard commented Nov 30, 2023 via email

@brettforbes
Copy link

Thanks for the cogent response. It turns out that i was just being paranoid then, my bad. I have just been getting concerned about the amount of capabilities we built our stack on that may be being nerfed in v3.0. I am glad this cardinality one is a non-issue and i really appreciate your response putting me at ease.
Thank you

@brettforbes
Copy link

brettforbes commented Apr 30, 2024

Hi All,

After speaking to Haikal, there are good reasons to move from the Concept API to Fetch, particularly speed. At present it takes between 2-5 secs to retrieve an object from TypeDB and transpile it to valid Stix JSON. This is mainly due to all of the network roundtrips that have to be done, so clearly one fetch query will be more effective.

The advantage of our current system is it is shape-based, so I can handle all JSON objects using the same ORM, the disadvantage is speed.

The new approach does mean a lot more code, since we have to build quite long Fetch statements for each individual object (e.g. 16-44) lines for each of our 85 objects, and then build the transpile code (from returned Fetch JSON to Stix JSON). This figure assumes a single main object, 4 lines, and then 3-11 optional sub objects with relations, with 4 lines each, if we use the class hierarchy. But the benefit will be far greater speed, totally agreed.

We probably wont be able to make this move for some months, due to resourcing, but we agree it will be worth it. At the same time we can update our 2.500 lines of schema code to v3. This will place us in good position to add on another 50-80 cybersecurity objects (e.g. SBOM's, Vulnerabilities, Risk etc.)

Onwards and upwards for TypeDB and our cybersecurity application!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants