From e1ac1dff791331ade147411821fa358c2dd45a82 Mon Sep 17 00:00:00 2001 From: Jonathan Zollinger <62955101+Jonathan-Zollinger@users.noreply.github.com> Date: Mon, 4 Sep 2023 21:08:43 -0600 Subject: [PATCH 01/13] Docs(import data): fix small typo (#641) ## Description This is just a simple typo I found. --- content/howto/importdata/live-loader.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/howto/importdata/live-loader.md b/content/howto/importdata/live-loader.md index 0790e7ca..d1b17e27 100644 --- a/content/howto/importdata/live-loader.md +++ b/content/howto/importdata/live-loader.md @@ -24,7 +24,7 @@ Those files have been generated by an [export]({{< relref "about-export.md" >}}) 1. Obtain dgraph binary or the latest docker image by following the [installation]({{< relref "download.md" >}}) instructions. This is required to run Dgraph CLI command `dgraph live`. -1. Obtain the `GRPC endpoint` of you Dgraph Cloud backend and a valid `Client API key`. +1. Obtain the `GRPC endpoint` of your Dgraph Cloud backend and a valid `Client API key`. An administrator gets those information with the following steps: 1. Log into the Dgraph Cloud account, select the backend From d4e08b122181706c66f8e579f968c4c506bd168e Mon Sep 17 00:00:00 2001 From: Raphael Date: Thu, 21 Sep 2023 18:02:00 -0700 Subject: [PATCH 02/13] update graphql subscription doc (#645) add example using URQL and subscriptions-transport-ws --- .../directives/directive-withsubscription.md | 27 +++ content/graphql/subscriptions/index.md | 188 +++++++++++------- 2 files changed, 148 insertions(+), 67 deletions(-) create mode 100644 content/graphql/schema/directives/directive-withsubscription.md diff --git a/content/graphql/schema/directives/directive-withsubscription.md b/content/graphql/schema/directives/directive-withsubscription.md new file mode 100644 index 00000000..f8f959dd --- /dev/null +++ b/content/graphql/schema/directives/directive-withsubscription.md @@ -0,0 +1,27 @@ ++++ +title = "@withSubscription" +weight = 7 +[menu.main] + parent = "directives" + identifier = "directive-withsubscription" ++++ + + +The `@withSubscription` directive enables **subscription** operation on a GraphQL type. + +A subscription notifies your client with changes to back-end data using the WebSocket protocol. +Subscriptions are useful to get low-latency, real-time updates. + +To enable subscriptions on any type add the `@withSubscription` directive to the schema as part of the type definition, as in the following example: + +```graphql +type Todo @withSubscription { + id: ID! + title: String! + description: String! + completed: Boolean! +} +``` + +Refer to [GraphQL Subscriptions]({{< relref "graphql/subscriptions" >}}) to learn how to use subscriptions in you client application. + diff --git a/content/graphql/subscriptions/index.md b/content/graphql/subscriptions/index.md index d24106b0..57cdf771 100644 --- a/content/graphql/subscriptions/index.md +++ b/content/graphql/subscriptions/index.md @@ -15,7 +15,7 @@ The client can unsubscribe by sending a message to the server. The server can al ![Subscription](/images/graphql/subscription_flow.png "Subscription in GraphQL") -## How to enable subscriptions in GraphQL +## Enable subscriptions in GraphQL In GraphQL, it's straightforward to enable subscriptions on any type. You can add the `@withSubscription` directive to the schema as part of the type definition, as in the following example: @@ -28,52 +28,12 @@ type Todo @withSubscription { } ``` -### Example +## @withSubscription with @auth -After updating the schema with the `@withSubscription` directive, you can execute a subscription query and receive updates when the subscription query result is updated, as follows: +You can use [@auth]({{< relref "graphql/schema/directives/auth">}}) access control rules in conjunction with `@withSubscription`. -![Subscription](/images/graphql/subscription_example.gif "Subscription Example") -## Apollo client setup - -To learn about using subscriptions with Apollo client, see a blog post on [GraphQL Subscriptions with Apollo client](https://dgraph.io/blog/post/how-does-graphql-subscription/). - -## Subscriptions to custom DQL - -You can use the `@withSubscription` directive on GraphQL types to generate subscription queries for that type. -You can also apply this directive to custom DQL queries by specifying `@withSubscription` on individual DQL queries in `type Query`, -and those queries will be added to `type subscription`. - -{{% notice "note" %}} -Currently, Dgraph only supports subscriptions on custom DQL queries. So, you -can't subscribe to custom HTTP queries. -{{% /notice %}} - -For example, see the custom DQL query `queryUserTweetCounts` below: - -```graphql -type Query { - queryUserTweetCounts: [UserTweetCount] @withSubscription @custom(dql: """ - query { - queryUserTweetCounts(func: type(User)) { - screen_name: User.screen_name - tweetCount: count(User.tweets) - } - } - """) -} -``` - -Because the `queryUserTweetCounts` query has a `@withSubscription` directive, it -will be added to the `subscription` type, allowing users to subscribe to this query. - -## Authorization with subscriptions - -Authorization adds more power to GraphQL subscriptions. You can use all of the authorization features that are available when running queries. -Additionally, you can specify when the subscription automatically terminates (the "timeout" of the subscription) in the JWT. - -### Schema -Consider following Schema that has both the `@withSubscription` and `@auth` directives defined on type `Todo`. The authorization rule enforces that only to-do tasks owned by `$USER` (defined in the JWT) are visible. +Consider following Schema that has both the `@withSubscription` and `@auth` directives defined on type `Todo`. ```graphql type Todo @withSubscription @auth( @@ -89,48 +49,142 @@ type Todo @withSubscription @auth( text: String! @search(by: [term]) owner: String! @search(by: [hash]) } -# Dgraph.Authorization {"VerificationKey":"secret","Header":"Authorization","Namespace":"https://dgraph.io","Algo":"HS256"} +# Dgraph.Authorization {"Header":"X-Dgraph-AuthToken","Namespace":"https://dgraph.io/jwt/claims","jwkurl":"https://xyz.clerk.accounts.dev/.well-known/jwks.json","audience":["dgraph"],"ClosedByDefault":true} +``` +The generated GraphQL API expects a JWT token in the `X-Dgraph-AuthToken` header and uses the `USER` claim to apply a rule based access control (RBAC): the authorization rule enforces that only to-do tasks owned by `$USER` are returned. + + +## WebSocket client +Dgraph uses the websocket subprotocol `subscription-transport-ws`. + +Clients must be instantiated using the WebSocket URL of the GraphQL API which is your [Dgraph GraphQL endpoint]({{< relref "graphql/graphql-clients/endpoint/_index.md">}}) with ``https`` replaced by ``wss``. + +If your Dgraph endpoint is ``https://blue-surf-0033.us-east-1.aws.cloud.dgraph.io/graphql`` +the WebSocket URL is ``wss://blue-surf-0033.us-east-1.aws.cloud.dgraph.io/graphql`` + +If your GraphQL API is configured to expect a JWT token in a header, you must configure the WebSocket client to pass the token. Additionally, the subscription terminates when the JWT expires. + + +Here are some examples of frontend clients setup. + +### URQL client setup in a React application + +In this scenario, we are using [urql client](https://formidable.com/open-source/urql/) and `subscriptions-transport-ws` modules. + +In order to use a GraphQL subscription query in a component, you need to +- instantiate a subscriptionClient +- instantiate a URQL client with a 'subscriptionExchange' using the subscriptionClient + +```js +import { Client, Provider, cacheExchange, fetchExchange, subscriptionExchange } from 'urql'; +import { SubscriptionClient } from 'subscriptions-transport-ws'; + + const subscriptionClient = new SubscriptionClient( + process.env.REACT_APP_DGRAPH_WSS, + { reconnect: true, + connectionParams: {"X-Dgraph-AuthToken" : props.token} + } + ); + + const client = new Client({ + url: process.env.REACT_APP_DGRAPH_ENDPOINT, + fetchOptions: { headers: { "X-Dgraph-AuthToken": `Bearer ${props.token}` } }, + exchanges: [ + cacheExchange, + fetchExchange, + subscriptionExchange({ + forwardSubscription: request => subscriptionClient.request(request), + }) + ]}) + ``` + +In this example, + +- **process.env.REACT_APP_DGRAPH_ENDPOINT** is your [Dgraph GraphQL endpoint]({{< relref "graphql/graphql-clients/endpoint/_index.md">}}) +- **process.env.REACT_APP_DGRAPH_WSS** is the WebSocket URL +- **props.token** is the JWT token of the logged-in user. + +Note that we are passing the JWT token in the GraphQL client using 'fetchOptions' and in the WebSocket client using 'connectionParams'. + +Assuming we are using graphql-codegen, we can define a subcription query: +```js +import { graphql } from "../gql"; + +export const TodoFragment = graphql(` + fragment TodoItem on Todo { + id + text + } +`) + + +export const TodoSubscription = graphql(` + subscription myTodo { + queryTodo(first:100) { + ...TodoItem + } + } +`) ``` +and use it in a React component +```js +import { useQuery, useSubscription } from "urql"; +... +const [messages] = useSubscription({ query: MyMessagesDocument}); -### JWT +``` +That's it, the react component is able to use ``messages.data.queryTodo`` to display the updated list of Todos. -The subscription requires a JWT that declares the `$USER`, expiry, and other variables. -The JWT is passed from the GraphQL client as key-value pair, where the key is the `Header` given in the schema and the value is the JWT. -In the example below, the key is `Authorization` and the value is the JWT. -Most GraphQL clients have a separate header section to pass Header-JWT key-value pairs. In the Apollo client, these are passed -using `connectionParams`, as follows. +### Apollo client setup + +To learn about using subscriptions with Apollo client, see a blog post on [GraphQL Subscriptions with Apollo client](https://dgraph.io/blog/post/how-does-graphql-subscription/). + +To pass the user JWT token in the Apollo client,use `connectionParams`, as follows. ```javascript const wsLink = new WebSocketLink({ uri: `wss://${ENDPOINT}`, options: { reconnect: true, - connectionParams: { "Authorization": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJleHAiOjE2OTAxMjg2MjIsImh0dHBzOi8vZGdyYXBoLmlvIjp7IlJPTEUiOiJVU0VSIiwiVVNFUiI6IkFsaWNlIn0sImlzcyI6InRlc3QifQ.6AODlumsk9kbnwZHwy08l40PeqEmBHqK4E_ozNjQpuI", },}); + connectionParams: { "
": "", },}); ``` -{{% notice "note" %}} -Starting in release v21.03, Dgraph supports compression for subscriptions. -Dgraph uses `permessage-deflate` compression if the GraphQL client's -`Sec-Websocket-Extensions` request header includes `permessage-deflate`, as follows: -`Sec-WebSocket-Extensions: permessage-deflate`. -{{% /notice %}} +Use the header expected by the Dgraph.Authorization configuration of your GraphQL schema. -### Example +## Subscriptions to custom DQL -The following example shows the operation of subscriptions with authentication rules for the schema given above. +You can also apply `@withSubscription` directive to custom DQL queries by specifying `@withSubscription` on individual DQL queries in `type Query`, +and those queries will be added to `type subscription`. -First, you can generate the JWT as shown in the following image with expiry and `$USER` (the owner of a to-do task). -You can generate the JWT from [jwt.io](https://jwt.io/). The client should send the JWT to the server along with the request, as discussed above. +For example, see the custom DQL query `queryUserTweetCounts` below: -![Subscription-Generating-JWT](/images/graphql/Generating-JWT.png "Subscription with Auth Example") +```graphql +type Query { + queryUserTweetCounts: [UserTweetCount] @withSubscription @custom(dql: """ + query { + queryUserTweetCounts(func: type(User)) { + screen_name: User.screen_name + tweetCount: count(User.tweets) + } + } + """) +} +``` -Next, Dgraph runs the subscription and send updates. You can see that only the to-do tasks that were added with the owner name "Alice" are visible in the subscription. +`queryUserTweetCounts` is added to the `subscription` type, allowing users to subscribe to this query. -![Subscription+Auth-Action](/images/graphql/Auth-Action.gif "Subscription with Auth Example") +{{% notice "note" %}} +Currently, Dgraph only supports subscriptions on custom **DQL queries**. You +can't subscribe to custom **HTTP queries**. +{{% /notice %}} -Eventually, the JWT expires and the subscription terminates, as shown below. -![Subscription+Timeout](/images/graphql/Subscription-Timeout.gif "Subscription with Auth Example") +{{% notice "note" %}} +Starting in release v21.03, Dgraph supports compression for subscriptions. +Dgraph uses `permessage-deflate` compression if the GraphQL client's +`Sec-Websocket-Extensions` request header includes `permessage-deflate`, as follows: +`Sec-WebSocket-Extensions: permessage-deflate`. +{{% /notice %}} From 0a7daeaf64bfc2f358576c8c4f7e4f4bd51826d6 Mon Sep 17 00:00:00 2001 From: Raphael Date: Thu, 21 Sep 2023 19:17:28 -0700 Subject: [PATCH 03/13] add Clerk.com config example (#646) Add an example of Dgraph.Authorization configuration with Clerk.com auth provider. --- content/graphql/security/jwt.md | 57 ++++++++++++++++++++++++++++++--- 1 file changed, 52 insertions(+), 5 deletions(-) diff --git a/content/graphql/security/jwt.md b/content/graphql/security/jwt.md index 7a490d05..75670cd4 100644 --- a/content/graphql/security/jwt.md +++ b/content/graphql/security/jwt.md @@ -11,7 +11,7 @@ When deploying a GraphQL schema, the admin user can set a ``# Dgraph.Authorizat This line must start with the exact string ``# Dgraph.Authorization`` and be at the bottom of the schema file. -### Configure JWT token handling +## Configure JWT token handling To configure how Dgraph should handle JWT token for ``/graphql`` endpoint : 1. Add a line starting with ``# Dgraph.Authorization`` and with the following parameters at the very end of your GraphQL schema. @@ -51,8 +51,51 @@ See the [RBAC rules]({{< relref "RBAC-rules.md">}}) and [Graph traversal rules]( ### Require JWT token To not only accept but to require the JWT token regardless of @auth directives in your GraphQL schema, set option "ClosedByDefault" to true in the `# Dgraph.Authorization` line. -### Examples +## Working with Authentication providers +Dgraph.Authorization is fully configurable to work with various authentication providers. +Authentication providers have options to configure how to generate JWT tokens. +Here are some configuration examples. + +### Clerk.com + +In your clerk dashboard, Access `JWT Templates` and create a template for Dgraph. + +Your template must have an `aud` (audience), this is mandatory for Dgraph when the token is verified using JWKURL. + +Decide on a claim namespace and add the information you want to use in your RBAC rules. + +We are using 'https://dgraph.io/jwt/claims' namespace in this example and have decided to get the user current organization, role ( clerk has currently two roles 'admin' and 'basic_member') and email. + +This is our JWT Template in Clerk: +```json +{ + "aud": "dgraph", + "https://dgraph.io/jwt/claims": { + "org": "{{org.name}}", + "role": "{{org.role}}", + "userid": "{{user.primary_email_address}}" + } +} +``` + +In the same configuration panel +- set the **token lifetime** +- copy the **JWKS Endpoint** + +Configure your Dgraph GraphQL schema with the following authorization +``` +# Dgraph.Authorization {"header":"X-Dgraph-AuthToken","namespace":"https://dgraph.io/jwt/claims","jwkurl":"https://<>.clerk.accounts.dev/.well-known/jwks.json","audience":["dgraph"],"closedbydefault":true} +``` +Note that +- **namespace** matches the namespace used in the JWT Template +- **audience** is an array and contains the **aud** used in the JWT token +- **jwkurl** is the **JWKS Endpoint** from Clerk + +You can select the header to receive the JWT token from your client app, `X-Dgraph-AuthToken` is a header authorized by default by Dgraph GraphQL API to pass CORS requirements. + + +## Other Dgraph.Authorization Examples To use a single JWK URL: @@ -82,7 +125,7 @@ Using HMAC-SHA256 token in `X-My-App-Auth` header and authorization claims in `h ### JWT format The value of the JWT ``header`` is expected to be in one of the following forms: -* Just the JWT token. +* Bare token. For example: ``` eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyLCJodHRwczovL215LmFwcC5pby9qd3QvY2xhaW1zIjp7fX0.Pjlxpf-3FhH61EtHBRo2g1amQPRi0pNwoLUooGbxIho @@ -96,7 +139,7 @@ The value of the JWT ``header`` is expected to be in one of the following forms: ### Error handling -If ClosedByDefault is set to true, and the JWT is not present or if the JWT token does not include the proper audience information, or is not properly encoded, Dgraph replies to requests on `/graphql` endpoint with an error message rejecting the operation similar to: +If ClosedByDefault is set to true, and the JWT is not present or if the JWT token does not include the proper audience information, or is not properly encoded, or is expired, Dgraph replies to requests on `/graphql` endpoint with an error message rejecting the operation similar to: ``` { "errors": [ @@ -111,4 +154,8 @@ If ClosedByDefault is set to true, and the JWT is not present or if the JWT toke "queryContact": [] },... ``` - +**Error messages** +- "couldn't rewrite query queryContact because a valid JWT is required but was not provided" +- "couldn't rewrite query queryMessage because unable to parse jwt token:token is expired by 5h49m46.236018623s" +- "couldn't rewrite query queryMessage because JWT `aud` value doesn't match with the audience" +- "couldn't rewrite query queryMessage because unable to parse jwt token:token signature is invalid" From af616744fa7f57f9fbbf663b59d121f6f33b4ac6 Mon Sep 17 00:00:00 2001 From: ShivajiKharse <115525374+shivaji-dgraph@users.noreply.github.com> Date: Fri, 22 Sep 2023 10:57:10 +0530 Subject: [PATCH 04/13] docs(dql):Add docs for unique directive (#644) --- content/dql/dql-schema.md | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/content/dql/dql-schema.md b/content/dql/dql-schema.md index fe3bfbb6..c55ca531 100644 --- a/content/dql/dql-schema.md +++ b/content/dql/dql-schema.md @@ -187,8 +187,28 @@ Query: } ``` +### Unique Directive +The unique constraint enables us to guarantee that all values of a predicate are distinct. To implement the @unique +directive for a predicate, you should define it in the schema and create an index on the predicate based on its type. +If a user does not add the proper index to the predicate, then Dgraph will return an error. + +Dgraph will automatically include the @upsert directive for the predicate. To enforce this uniqueness constraint, +a predicate must have an index, as explained below. Currently, we only support the @unique directive on newly created +predicates with the data types string and integer. + +| Data Type | Index | +| -------- | ------- | +| string | hash, exact | +| int | int | + +This is how you define the unique directive for a predicate. + +``` +email: string @unique @index(exact) . +``` + ### Upsert directive From bc16536a9d8a31c563f608893359013c56c0feac Mon Sep 17 00:00:00 2001 From: ShivajiKharse <115525374+shivaji-dgraph@users.noreply.github.com> Date: Fri, 22 Sep 2023 12:36:23 +0530 Subject: [PATCH 05/13] Docs(GraphQL):add docs for restoreTenant API (#647) --- content/enterprise-features/binary-backups.md | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/content/enterprise-features/binary-backups.md b/content/enterprise-features/binary-backups.md index cfef7966..44fc0999 100644 --- a/content/enterprise-features/binary-backups.md +++ b/content/enterprise-features/binary-backups.md @@ -648,6 +648,51 @@ mutation{ } } ``` + +## Namespace Aware Restore + +You can use namespace-aware restore to restore a single namespace from a backup that contains multiple namespaces. +The created restore will be available in the default namespace. For example, if you restore namespace 2 using the +restoreTenant API, then after the restore operation is completed, the cluster will have only the default namespace, +and it will contain the data from namespace 2. Namespace aware restore supports incremental restore. + +To restore from a backup to a live cluster, execute a mutation on the `/admin` endpoint with the following format: + +```graphql +mutation { + restoreTenant( + input: { + restoreInput: { + incrementalFrom: "incremental_backup_from" + location: "/path/to/backup/directory" + backupId: "id_of_backup_to_restore" + } + fromNamespace: namespaceToBeRestored + } + ) { + message + code + } +} +``` + +Documentation of restoreTenant inputs + +``` +input RestoreTenantInput { + """ + restoreInput contains fields that are required for the restore operation, + i.e., location, backupId, and backupNum + """ + restoreInput: RestoreInput + + """ + fromNamespace is the namespace of the tenant that needs to be restored into namespace 0 of the new cluster. + """ + fromNamespace: Int! +} +``` + ## Offline restore The restore utility is now a standalone tool. A new flag, `--encryption key-file=value`, is now part of the restore utility, so you can use it to decrypt the backup. The file specified using this flag must contain the same key that was used for encryption during backup. Alternatively, starting with `v20.07.0`, the `vault` superflag can be used to restore a backup. From 1d30f78806cf1be23c86d4596ee9734688058a56 Mon Sep 17 00:00:00 2001 From: Raphael Date: Fri, 22 Sep 2023 08:26:15 -0700 Subject: [PATCH 06/13] Update RBAC-rules.md (#648) Clarify that a claim can be an array of strings in RBAC rules. --- content/graphql/security/RBAC-rules.md | 48 ++++++++++++++++++++++---- 1 file changed, 41 insertions(+), 7 deletions(-) diff --git a/content/graphql/security/RBAC-rules.md b/content/graphql/security/RBAC-rules.md index 3741186d..643d769a 100644 --- a/content/graphql/security/RBAC-rules.md +++ b/content/graphql/security/RBAC-rules.md @@ -26,16 +26,51 @@ type User @auth( delete: ... ) ``` +RBAC rule supports ``eq`` or ``in`` functions to test the value of a [JWT claim]({{< relref "graphql/security/_index.md#jwt-claims" >}}) from the JWT token payload. -Where `` is a [JWT claim]({{< relref "graphql/security/_index.md#jwt-claims" >}}) from the JWT token payload. - -You can use ``eq`` or ``in`` function to test the value of any claim. +The claim value may be a string or array of strings. +For example the following schema has a @auth directive specifying that a delete operation on a User object can only be done if the connected user has a 'ROLE' claim in the JWT token with the value "admin" : +```graphql +type User @auth( + delete: { rule: "{$ROLE: { eq: \"admin\" } }"} + ) { + username: String! + @id todos: [Todo] +} +``` +The following JWT token payload will pass the test (provided that Dgraph.Authorization is configured correctly with the right namespace) +```json +{ + "aud": "dgraph", + "exp": 1695359621, + "https://dgraph.io/jwt/claims": { + "ROLE": "admin", + "USERID": "testuser@dgraph.io" + }, + "iat": 1695359591, + ... +} +``` +The rule is also working with an array of roles in the JWT token: +```json +{ + "aud": "dgraph", + "exp": 1695359621, + "https://dgraph.io/jwt/claims": { + "ROLE": ["admin","user"] + "USERID": "testuser@dgraph.io" + }, + "iat": 1695359591, + ... +} +``` +In the case of an array used with the "in" function, the rule is valid is at least one of the claim value is "in" the provided list. -For example the following schema has a @auth directive specifying that a delete operation on a User object can only be done in the connected user has a 'ROLE' claim in the JWT token with the value "ADMIN" : +For example, with the following rule, the previous token will be valid because one of the ROLE is in the authorized roles. ```graphql type User @auth( - delete: { rule: "{$ROLE: { eq: \"ADMIN\" } }"} + delete: { rule: "{$ROLE: { in: [\"admin\",\"superadmin\"] } }"} ) { username: String! @id todos: [Todo] @@ -70,12 +105,11 @@ For example, given the following JWT payload "https://xyz.io/jwt/claims": [ "ROLE": "ADMIN" ], - "USERROLE": "user1", "email": "random@example.com" } ``` -The authorization rules can use ``$ROLE`` (if `https://xyz.io/jwt/claims` is declared as the namespace to use ) and also ``$USERROLE`` or ``$email``. +If `https://xyz.io/jwt/claims` is declared as the namespace to use, the authorization rules can use ``$ROLE`` but also ``$email``. In cases where the same claim is present in the namespace and at the root level, the claim value in the namespace takes precedence. From 23f05b345f52491944e2f06c79bdc3285859a154 Mon Sep 17 00:00:00 2001 From: Ryan Fox-Tyler <60440289+ryanfoxtyler@users.noreply.github.com> Date: Fri, 5 Jan 2024 15:20:48 -0500 Subject: [PATCH 07/13] Update release table (#652) Updating the release table in docs to: 1. include the latest release (v23.1) 2. make the table more maintainable/default current 3. omit patch releases --- content/releases/index.md | 46 ++++++++++++++++++++------------------- 1 file changed, 24 insertions(+), 22 deletions(-) diff --git a/content/releases/index.md b/content/releases/index.md index 12f2c8b1..f8a10f11 100644 --- a/content/releases/index.md +++ b/content/releases/index.md @@ -6,7 +6,7 @@ title = "Dgraph Releases" weight = 14 +++ -The latest Dgraph release is the v23.00 series. +The latest Dgraph release is the v23.1 series. Dgraph releases starting v22.0.0 is following semver [See the post here](https://discuss.dgraph.io/t/dgraph-v22-0-0-rc1-20221003-release-candidate/). @@ -18,25 +18,27 @@ To learn about the latest releases and other important announcements, watch the ## Release series - Dgraph Release Series | Current Release | Supported? | First Release Date | End of life ------------------------|-----------------|------------|--------------------|-------------- - v23.0.x | [v23.0.0][] | Yes | May 2023 | N/A - v22.0.x | [v22.0.2][] | Yes | October 2022 | N/A - v21.12.x(discontinued)| v21.12.0 | No | December 2021 | December 2022 - v21.03.x | [v21.03.0][] | Yes | March 2021 | June 2023 - v20.11.x | [v20.11.0][] | No | December 2020 | December 2021 - v20.07.x | [v20.07.3][] | No | July 2020 | July 2021 - v20.03.x | [v20.03.7][] | No | March 2020 | March 2021 - v1.2.x | [v1.2.8][] | No | January 2020 | January 2021 - v1.1.x | [v1.1.1][] | No | January 2020 | January 2021 - v1.0.x | [v1.0.18][] | No | December 2017 | March 2020 + Release | First Release Date | End of Life +-----------------------|--------------------|-------------- + [v23.1][] | October 2023 | April 2025 + [v23.0][] | May 2023 | November 2024 + [v22.0][] | October 2022 | April 2024 + v21.12 (discontinued) | December 2021 | December 2022 + [v21.03][] | March 2021 | June 2023 + [v20.11][] | December 2020 | December 2021 + [v20.07][] | July 2020 | July 2021 + [v20.03][] | March 2020 | March 2021 + [v1.2][] | January 2020 | January 2021 + [v1.1][] | January 2020 | January 2021 + [v1.0][] | December 2017 | March 2020 -[v23.0.0]: https://discuss.dgraph.io/t/dgraph-release-v23-0-0-is-now-generally-available/18634 -[v22.0.2]: https://discuss.dgraph.io/t/dgraph-release-v22-0-2-is-now-generally-available/18117 -[v21.03.0]: https://discuss.dgraph.io/t/release-notes-v21-03-0-resilient-rocket/13587 -[v20.11.0]: https://discuss.dgraph.io/t/release-notes-v20-11-0-tenacious-tchalla/11942 -[v20.07.3]: https://discuss.dgraph.io/t/dgraph-v20-07-3-release/12107 -[v20.03.7]: https://discuss.dgraph.io/t/dgraph-v20-03-7-release/12077 -[v1.2.8]: https://discuss.dgraph.io/t/dgraph-v1-2-8-release/11183 -[v1.1.1]: https://discuss.dgraph.io/t/dgraph-v1-1-1-release/5664 -[v1.0.18]: https://discuss.dgraph.io/t/dgraph-v1-0-18-release/5663 +[v23.1]: https://discuss.dgraph.io/t/dgraph-23-1-0-is-generally-available-on-dgraph-cloud-dockerhub-and-github/18980 +[v23.0]: https://discuss.dgraph.io/t/dgraph-release-v23-0-0-is-now-generally-available/18634 +[v22.0]: https://discuss.dgraph.io/t/dgraph-release-v22-0-2-is-now-generally-available/18117 +[v21.03]: https://discuss.dgraph.io/t/release-notes-v21-03-0-resilient-rocket/13587 +[v20.11]: https://discuss.dgraph.io/t/release-notes-v20-11-0-tenacious-tchalla/11942 +[v20.07]: https://discuss.dgraph.io/t/dgraph-v20-07-3-release/12107 +[v20.03]: https://discuss.dgraph.io/t/dgraph-v20-03-7-release/12077 +[v1.2]: https://discuss.dgraph.io/t/dgraph-v1-2-8-release/11183 +[v1.1]: https://discuss.dgraph.io/t/dgraph-v1-1-1-release/5664 +[v1.0]: https://discuss.dgraph.io/t/dgraph-v1-0-18-release/5663 From 1037845000da29edfd9ea6a2fe2c54b1cb8110dd Mon Sep 17 00:00:00 2001 From: Ryan Fox-Tyler <60440289+ryanfoxtyler@users.noreply.github.com> Date: Fri, 5 Jan 2024 15:33:10 -0500 Subject: [PATCH 08/13] Update build.sh Removing auto-version selection, as docs must be ready --- scripts/build.sh | 13 +------------ 1 file changed, 1 insertion(+), 12 deletions(-) diff --git a/scripts/build.sh b/scripts/build.sh index 05696293..cdba98af 100755 --- a/scripts/build.sh +++ b/scripts/build.sh @@ -31,20 +31,9 @@ THEME_BRANCH="${THEME_BRANCH:-main}" # and then the older versions in descending order, such that the # build script can place the artifact in an appropriate location. -getMajorVersions=$(curl -s https://get.dgraph.io/latest \ -| grep -o '"majorReleases":.*]' | grep -o '".*"' | grep -o '"[^[]*$' \ -| sed "s/\"//g" | sed "s/\,/ /g" | sed "s/v20.03/ /g" | sed "s/v21.12/ /g" | sed "s/v20.07/ /g") - -MAJOR_VERSIONS=( - $getMajorVersions -) - VERSIONS_ARRAY=( + 'v23.0' 'v22.0' - ${MAJOR_VERSIONS:0} - ${MAJOR_VERSIONS[@]:1} - 'main' - 'v21.12' ) joinVersions() { From 6813c7a2a7c37688d957dd32f94f6371e3b3726f Mon Sep 17 00:00:00 2001 From: Ryan Fox-Tyler <60440289+ryanfoxtyler@users.noreply.github.com> Date: Fri, 5 Jan 2024 15:51:37 -0500 Subject: [PATCH 09/13] Update build.sh (#654) Undo change --- scripts/build.sh | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/scripts/build.sh b/scripts/build.sh index cdba98af..05696293 100755 --- a/scripts/build.sh +++ b/scripts/build.sh @@ -31,9 +31,20 @@ THEME_BRANCH="${THEME_BRANCH:-main}" # and then the older versions in descending order, such that the # build script can place the artifact in an appropriate location. +getMajorVersions=$(curl -s https://get.dgraph.io/latest \ +| grep -o '"majorReleases":.*]' | grep -o '".*"' | grep -o '"[^[]*$' \ +| sed "s/\"//g" | sed "s/\,/ /g" | sed "s/v20.03/ /g" | sed "s/v21.12/ /g" | sed "s/v20.07/ /g") + +MAJOR_VERSIONS=( + $getMajorVersions +) + VERSIONS_ARRAY=( - 'v23.0' 'v22.0' + ${MAJOR_VERSIONS:0} + ${MAJOR_VERSIONS[@]:1} + 'main' + 'v21.12' ) joinVersions() { From 67d43538abef3320f1d2a1fb063122f868dc7589 Mon Sep 17 00:00:00 2001 From: Ryan Fox-Tyler <60440289+ryanfoxtyler@users.noreply.github.com> Date: Tue, 16 Jan 2024 09:02:16 -0500 Subject: [PATCH 10/13] Update CODEOWNERS (#656) Updating codeowners to map to GitHub Team for easier management --- .github/CODEOWNERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 12bc66c1..a35148a3 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1,4 +1,4 @@ # CODEOWNERS info: https://help.github.com/en/articles/about-code-owners # Owners are automatically requested for review for PRs that changes code # that they own. -* @rderbier @MichelDiz @damonfeldman @rarvikar @Rajakavitha1 +* @dgraph-io/committers @rderbier From f6301173b1fc6139244466921798c14a9d1313aa Mon Sep 17 00:00:00 2001 From: rderbier Date: Tue, 16 Apr 2024 10:53:43 -0700 Subject: [PATCH 11/13] Update README.md --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index 83c2ff24..a3cdd76e 100644 --- a/README.md +++ b/README.md @@ -102,3 +102,6 @@ Pass custom Go-GRPC example to the runnable by passing a `customExampleGoGRPC` t ``` **Note:** Runnable doesn't support passing a multiline string as an argument to a shortcode. Therefore, you have to create the whole custom example in a single line string by replacing newlines with `\n`. + +## History +add Hypermode banner by updating the hugo-docs repository with topbat template. \ No newline at end of file From d10ab88d7ae0d8998cc938d6454798c0cb265ef3 Mon Sep 17 00:00:00 2001 From: Gajanan Date: Tue, 23 Apr 2024 14:01:29 -0700 Subject: [PATCH 12/13] Add DQL additions for supporting vector type. New datatype has been added from 24.0 for supporting vector tyoe and semantic search. --- content/dql/dql-schema.md | 9 +++++++++ content/dql/predicate-indexing.md | 6 ++++++ content/query-language/functions.md | 14 ++++++++++++++ 3 files changed, 29 insertions(+) diff --git a/content/dql/dql-schema.md b/content/dql/dql-schema.md index c55ca531..182f08d3 100644 --- a/content/dql/dql-schema.md +++ b/content/dql/dql-schema.md @@ -16,6 +16,9 @@ revenue: float . running_time: int . starring: [uid] . director: [uid] . +description: string . + +description_vector: float32vector @index(hnsw(metric:"cosine")) . type Person { name @@ -28,6 +31,8 @@ type Film { running_time starring director + description + description_vector } ``` @@ -112,6 +117,10 @@ For all triples with a predicate of scalar types the object is a literal. are RFC 3339 compatible which is different from ISO 8601(as defined in the RDF spec). You should convert your values to RFC 3339 format before sending them to Dgraph.{{% /notice %}} +### Vector Type + +The `float32vector` type denotes a vector of floating point numbers used to store embeddings of one or more predicates or node embeddings. A schema type can contain more than one vector. + ### UID Type The `uid` type denotes a relationship; internally each node is identified by it's UID which is a `uint64`. diff --git a/content/dql/predicate-indexing.md b/content/dql/predicate-indexing.md index 2f7669c4..c8973a80 100644 --- a/content/dql/predicate-indexing.md +++ b/content/dql/predicate-indexing.md @@ -34,6 +34,12 @@ transaction conflict rate. Use only the minimum number of and simplest indexes that your application needs. {{% /notice %}} +## Vector Indices + +For fast semantic search `hsnw` (**Hierarchical Navigable Small World**) index is available on `float32vector`. The index is created based on one of the following distance metrics: `cosine`, `euclidean`, and `dotproduct`. + +Note that vector index must be defined using `@index` directive: `description_vector: float32vector @index(hnsw(metric:"cosine")) .` + ## DateTime Indices The indices available for `dateTime` are as follows. diff --git a/content/query-language/functions.md b/content/query-language/functions.md index f05f0f94..39b3f005 100644 --- a/content/query-language/functions.md +++ b/content/query-language/functions.md @@ -177,6 +177,20 @@ Same query with a Levenshtein distance of 3. } {{< /runnable >}} +## Similarity (aka Sematic) Search + +Syntax Examples: `similar_to(description_vector, 3, "[0.9, 0.8, 0, 0]")` + +Alternatively the vector can be passed as a variable: `similar_to(description_vector, 3, $vec)` + +This function finds the objects that have description_vector similar to the input vector. The second parameter, `3` specifies that top 3 matches be returned. + +Schema Types: `float32vector` + +Index Required: `hnsw` + +Similarity search results are based on distance metrics used to create index (`cosine`, `euclidean`, and `dotproduct`). The shorter distance indicates more similarity. + ## Full-Text Search Syntax Examples: `alloftext(predicate, "space-separated text")` and `anyoftext(predicate, "space-separated text")` From 85c1d94a2d43e393e063fc713c32316a2c5a41ec Mon Sep 17 00:00:00 2001 From: rderbier Date: Wed, 24 Apr 2024 11:15:10 -0700 Subject: [PATCH 13/13] corrections --- content/dql/dql-schema.md | 7 +++++- content/dql/predicate-indexing.md | 33 +++++++++++++++++++++++++++-- content/query-language/functions.md | 11 +++++----- 3 files changed, 43 insertions(+), 8 deletions(-) diff --git a/content/dql/dql-schema.md b/content/dql/dql-schema.md index 182f08d3..c52c2bfe 100644 --- a/content/dql/dql-schema.md +++ b/content/dql/dql-schema.md @@ -119,7 +119,12 @@ convert your values to RFC 3339 format before sending them to Dgraph.{{% /notice ### Vector Type -The `float32vector` type denotes a vector of floating point numbers used to store embeddings of one or more predicates or node embeddings. A schema type can contain more than one vector. +The `float32vector` type denotes a vector of floating point numbers, i.e an ordered array of float32. A node type can contain more than one vector predicate. + +Vectors are normaly used to store embeddings obtained from other information through an ML model. When a `float32vector` is [indexed]({{}}), the DQL [similar_to]({{}}) function can be used for similarity search. + + + ### UID Type diff --git a/content/dql/predicate-indexing.md b/content/dql/predicate-indexing.md index c8973a80..17daa541 100644 --- a/content/dql/predicate-indexing.md +++ b/content/dql/predicate-indexing.md @@ -9,6 +9,15 @@ weight = 4 Filtering on a predicate by applying a [function]({{< relref "query-language/functions.md" >}}) requires an index. +Indices are defined in the [Dgraph types schema]({{}}) using `@index` directive. + +Here are some examples: +``` +name: string @index(term) . +release_date: datetime @index(year) . +description_vector: float32vector @index(hnsw(metric:"cosine")) . +``` + When filtering by applying a function, Dgraph uses the index to make the search through a potentially large dataset efficient. All scalar types can be indexed. @@ -17,6 +26,8 @@ Types `int`, `float`, `bool` and `geo` have only a default index each: with toke Types `string` and `dateTime` have a number of indices. +Type `float32vector` supports `hsnw` index. + ## String Indices The indices available for strings are as follows. @@ -36,9 +47,27 @@ that your application needs. ## Vector Indices -For fast semantic search `hsnw` (**Hierarchical Navigable Small World**) index is available on `float32vector`. The index is created based on one of the following distance metrics: `cosine`, `euclidean`, and `dotproduct`. +The indices available for `float32vector` are as follows. + +| Dgraph function | Required index / tokenizer | Notes | +| :----------------------- | :------------ | :--- | +| `similar_to` | `hsnw` | HSNW index supports parameters `metric` and `exponent`. | -Note that vector index must be defined using `@index` directive: `description_vector: float32vector @index(hnsw(metric:"cosine")) .` + +# + +`hsnw` (**Hierarchical Navigable Small World**) index supports the following parameters +- metric : indicate the metric to use to compute vector similarity. One of `cosine`, `euclidean`, and `dotproduct`. Default is `euclidean`. + +- exponent : An integer, represented as a string, roughly representing the number of vectors expected in the index in power of 10. The exponent value,is used to set "reasonable defaults" for HSNW internal tuning parameters. Default is "4" (10^4 vectors). + + +Here are some examples: +``` +simple_vector: float32vector @index(hnsw) . +description_vector: float32vector @index(hnsw(metric:"cosine")) . +large_vector: float32vector @index(hnsw(metric:"euclidean",exponent:"6")) . +``` ## DateTime Indices diff --git a/content/query-language/functions.md b/content/query-language/functions.md index 39b3f005..b410f922 100644 --- a/content/query-language/functions.md +++ b/content/query-language/functions.md @@ -177,19 +177,20 @@ Same query with a Levenshtein distance of 3. } {{< /runnable >}} -## Similarity (aka Sematic) Search +## Vector Similarity Search -Syntax Examples: `similar_to(description_vector, 3, "[0.9, 0.8, 0, 0]")` +Syntax Examples: `similar_to(predicate, 3, "[0.9, 0.8, 0, 0]")` -Alternatively the vector can be passed as a variable: `similar_to(description_vector, 3, $vec)` +Alternatively the vector can be passed as a variable: `similar_to(predicate, 3, $vec)` -This function finds the objects that have description_vector similar to the input vector. The second parameter, `3` specifies that top 3 matches be returned. +This function finds the nodes that have `predicate` close to the provided vector. The search is based on the distance metric specified in the index (`cosine`, `euclidean`, or `dotproduct`). The shorter distance indicates more similarity. +The second parameter, `3` specifies that top 3 matches be returned. Schema Types: `float32vector` Index Required: `hnsw` -Similarity search results are based on distance metrics used to create index (`cosine`, `euclidean`, and `dotproduct`). The shorter distance indicates more similarity. + ## Full-Text Search