Use the readTs for reads via LocalCache #6317
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
danielmai
pushed a commit
that referenced
this pull request
Sep 18, 2020
Use the readTs for reads via LocalCache. This would ensure that users don't get a lot of ReadTs less than MinTs errors. This PR also fixes the flaky test in increment_test.go. This has a side-effect on posting list cache. That cache would need to understand versioning to be effective. Co-authored-by: Daniel Mai <daniel@dgraph.io> (cherry picked from commit 2de4675)
parasssh
pushed a commit
that referenced
this pull request
Sep 18, 2020
Use the readTs for reads via LocalCache. This would ensure that users don't get a lot of ReadTs less than MinTs errors. This PR also fixes the flaky test in increment_test.go. This has a side-effect on posting list cache. That cache would need to understand versioning to be effective. Co-authored-by: Daniel Mai <daniel@dgraph.io> (cherry picked from commit 2de4675)
parasssh
pushed a commit
that referenced
this pull request
Sep 18, 2020
Use the readTs for reads via LocalCache. This would ensure that users don't get a lot of ReadTs less than MinTs errors. This PR also fixes the flaky test in increment_test.go. This has a side-effect on posting list cache. That cache would need to understand versioning to be effective. Co-authored-by: Daniel Mai <daniel@dgraph.io> (cherry picked from commit 2de4675)
parasssh
pushed a commit
that referenced
this pull request
Sep 18, 2020
Use the readTs for reads via LocalCache. This would ensure that users don't get a lot of ReadTs less than MinTs errors. This PR also fixes the flaky test in increment_test.go. This has a side-effect on posting list cache. That cache would need to understand versioning to be effective. Co-authored-by: Daniel Mai <daniel@dgraph.io> (cherry picked from commit 2de4675) Co-authored-by: Manish R Jain <manish@dgraph.io>
parasssh
pushed a commit
that referenced
this pull request
Sep 18, 2020
Use the readTs for reads via LocalCache. This would ensure that users don't get a lot of ReadTs less than MinTs errors. This PR also fixes the flaky test in increment_test.go. This has a side-effect on posting list cache. That cache would need to understand versioning to be effective. Co-authored-by: Daniel Mai <daniel@dgraph.io> (cherry picked from commit 2de4675) Co-authored-by: Manish R Jain <manish@dgraph.io>
Sankalan13
added a commit
that referenced
this pull request
Sep 21, 2020
* fix(docs): fix URL paths and make deploy page to not show blank page (#6239) * Fix URL paths (#6189) (cherry picked from commit 4ee4af8) * fix deploy page to not show blank page (cherry picked from commit 89e012b) Co-authored-by: Apoorv Vardhan <vardhanapoorv@users.noreply.github.com> * feat(docs): Port GraphQL docs to v20.07 (#6202) * Port GraphQL docs to v20.07 This reverts commit 8db61a3. Co-authored-by: Pawan Rawal <pawan0201@gmail.com> * doc: Setting the canonical path to each docs without version (#6249) Passing a few variables to the hugo theme * fix: remove unnecessary curly braces from doc (#6256) Co-authored-by: Sankalan Parajuli <sankalan.13@gmail.com> * fix(Dgraph): Stop forcing RAM mode for the write-ahead log. (#6142) (#6259) This change also adds a way to set the table and value log loading modes for the w directory independently of the values for the p directory. Fixes DGRAPH-1898. (cherry picked from commit 5f5aa9c) * Enabling Discuss comments * Fix(Dgraph): Fix how visited nodes are detected in recurse queries. (#6272) (#6276) Currently a node is marked as visited if it's been visited before AND has been visited from the same source UID. In dense graphs, the second condition leads to exponential growth of the data and to other issues such as responses that are too big to encode. Removing this condition fixes the issue. Fixed tests and verified the new output makes sense. Fixes DGRAPH-2337 (cherry picked from commit 3638c12) * fix panicwrap parent check (#6264) (#6299) (cherry picked from commit 52e136f) * release/v20.07 - add --cache_mb and --cache_percentage flag (#6286) * fix(Dgraph): Sort manifests by BackupNum in file handler. (#6263) (#6279) This was being done in the S3 handler but not in the file handler. By default the paths are sorted by the filename but sorting by backup number is more robust. (cherry picked from commit df63d6c) * fix(ACL Query): Fixes queries which use variable at the top level (#6290) (#6298) (cherry picked from commit 6f883a4) * Revert "Fix(Dgraph): Fix how visited nodes are detected in recurse queries. (#6272) (#6276)" (#6306) This reverts commit 9df9f92. * fix(Dgraph): Don't store start_ts in postings. (#6213) Clear out the start_ts field in the postings of deltas before they are marshalled to avoid storing that field in disk. This field is only meant to be used during in-memory processing. Related to https://discuss.dgraph.io/t/start-ts-not-being-cleared-before-postings-are-written-to-disk/9146 (cherry picked from commit fbbd731) * docs: add info about __typename in custom DQL (#6248) (#6273) Reference: [Discuss Post](https://discuss.dgraph.io/t/custom-dql-typename-bug-strange-behaviour/9341) (cherry picked from commit 0faae40) * fix(GraphQL): fixes unexpected fragment behaviour (#6228) (#6274) Fixes: #5516 Fixes: [Discuss Issue](https://discuss.dgraph.io/t/graphql-fragments-generates-unexpected-behaviour/7470) (cherry picked from commit dcce8bf) * fix(task): Return error on closed DB (#6075) (#6320) Fixes DGRAPH-2181 The queries in dgraph are processed in separate goroutines. The pstore badger DB could be closed while the query was being processed. This causes panics such as ``` panic: runtime error: invalid memory address or nil pointer dereference panic: Unclosed iterator at time of Txn.Discard. [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x11d2311] goroutine 19298 [running]: github.com/dgraph-io/badger/v2.(*Txn).Discard(0xc05586bc20) /go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200718033852-37ee16d8ad1c/txn.go:517 +0xc1 panic(0x19e26a0, 0x2988560) /usr/local/go/src/runtime/panic.go:969 +0x166 github.com/dgraph-io/badger/v2/skl.(*Skiplist).IncrRef(...) ``` This PR attempts to reduce the number of such crashes. This PR doesn't fix the actual issue but it tries to reduce the probability of such crashes by checking if badger is not closed before accessing it. An ideal fix would be to stop all the goroutines started by dgraph while closing DB so that we don't try to read from a closed badger DB. (cherry picked from commit 3cea0fe) * fix(Dgraph): make backups cancel other tasks (#6152) (#6243) Backups should be added to the list of tasks (rollups, snapshots, etc) that are managed by Dgraph so that rollups and other tasks are paused during the backup. (cherry picked from commit ea15b66) * Update badger to v2.2007.2 (#6294) This badger update brings the following changes from badger. https://github.com/dgraph-io/badger/commits/b41e77a Fix(cleanup): Avoid truncating in value.Open on error https://github.com/dgraph-io/badger/commits/a0d4903 fix(cleanup): Do not close cache before compaction * posting list fixes. (#6303) (#6332) (cherry picked from commit deded8b) * fix: Online Restore honors credentials passed in (#6295) (#6302) (cherry picked from commit a8a6e85) Co-authored-by: Tejas Dinkar <tejas@gja.in> * (release/v20.07) Fix(Dgraph): Add a lock to backups to process one request at a time. (#6339) It's possible that two requests reach the server around the same time and send a requests to the alphas with the same backupNum. This could lead to issues further down the line. Related to DGRAPH-2295 (cherry picked from commit 5b79260) * (release/v20.07) fix(Dgraph): Add flags to set table and vlog loading mode for zero. (#6342) Related to DGRAPH-2189 (cherry picked from commit bf79999) * proto fix needed for PR #6331 (#6337) (#6346) (cherry picked from commit 844f34d) * chore(GraphQL): remove skipped tests (#6078) (#6347) Fixes DGRAPH-2357. Fixes GRAPHQL-562. (cherry picked from commit 91e28d5) * test: add --build option for docker-compose up in test script (#6348) (#6352) Fixes DGRAPH-2356 This PR fixes random GraphQL custom logic e2e test failures in TeamCity by requiring docker-compose to always build images before starting the containers, as the mock server in custom logic tests can change any time. (cherry picked from commit 536b317) * cherry-pick v20.07: fix(GraphQL): incorrect generatedSchema in updateGQLSchema (#6349) (#6354) * fix(GraphQL): incorrect generatedSchema in updateGQLSchema (#6349) This PR fixes the behaviour where Dgraph schema was being given as generatedSchema field in updateGQLSchema request. Now, the newly generated complete GraphQL schema is given as generatedSchema. (cherry picked from commit 0b8681c) # Conflicts: # graphql/e2e/schema/schema_test.go * (release/v20.07) fix(Dgraph): add flag to set up compression in zero. (#6355) (cherry picked from commit 254bd29) * fix(dgraph): sentry nil pointer check (#6372) (#6374) (cherry picked from commit 7857fa2) * doc: Adding a note on backend modes * doc: Adding a security page on slash * chore(Docs): Update graph-links.md to remove a duplicate word. (#6378) remove duplicate word "doesn't" from description of two-way edges. * doc: One-Click Documentation for Slash (#6381 #6383) (#6384) (cherry picked from commit e165710) * Use z.Closer instead of y.Closer (#6394) (#6399) The closer type was moved from badger/y to ristretto/z in dgraph-io/ristretto#191 (cherry picked from commit 0287838) * (release/v20.07) docs: Add section on automating backups. (#6405) (cherry picked from commit 0fab710) * relase/v20.07 - Fix(Alpha): MASA: Make Alpha Shutdown Again (#6313) (#6402) * Fix(Alpha): MASA: Make Alpha Shutdown Again (#6313) This PR removes the usage of context.Background() in groups.go and ensures that all the various goroutines exit as intended. Changes: * Shutdown SubscribeForUpdates correctly. * Fix up all the closing conditions. * Consolidate updaters into one closer * Update Badger to master * fix(build): Update ResetAcl args for OSS build. * chore: Remove TODO comment. Co-authored-by: Daniel Mai <daniel@dgraph.io> (cherry picked from commit f1941b3) * fix(shutdown): Force exit if CTRL-C is caught before initialization (#6359) (#6407) Forcefully kill alpha if CTRL-C is caught before node initialization completes. Fixes DGRAPH-2377 Co-authored-by: Manish R Jain <manish@dgraph.io> (cherry picked from commit 41c2052) * Update advanced-queries.md * fix(Query) Fix Star_All delete query when used with ACL enabled (#6331) (#6336) * docs: Add netlify deploy docs to slash docs (#6425) (#6428) (cherry picked from commit d94e5da) * docs: add whitelisting issue error in troubleshoot page to help the community. (#6422) Fixes DGRAPH-2317. Issue is with v20.07 we restrict the admin operations from alpha instances only. But this hasn't been mentioned during the deployment process in any examples. Hence community is running into issues like Unauthorized IP address X.X.X.X. This doc addition in troubleshoot page will help the community members if they run into this issue. * Add badger to docs (#6430) * fix(bulk): Batch list in bulk loader to avoid panic (#6446) * Changelog and Codename (#6488) * fix: Regenerate protos. The different generated proto file makes the release.sh script fail with "Generated protos different in release." Changes * Called "make regenerate" * fix(GraphQL): fix introspection completion bug (#6385) (#6389) Fixes GRAPHQL-673. This PR fixes the issue where introspection queries would break if there were two types implementing same interface and having fields with same name in those two types and that repeating field is also a field in introspection query. This was introduced after #6228. (cherry picked from commit 243a336) * fix(GraphQL): don't generate orderable enum value for list fields (#6392) (#6413) Fixes GRAPHQL-650. For this user schema: ``` type Starship { id: ID! name: String! @search(by: [term]) length: Int64 tags: [String] createdAt: DateTime } ``` we were generating complete GraphQL schema with StarshipOrderable as: ``` enum StarshipOrderable { name length tags createdAt } ``` that would cause a DQL query to be formed which errors out, see this GraphQL query: ``` query { queryStarship(order: {asc: tags}) { id name length tags } } ``` It gives back: ``` { "errors": [ { "message": "Dgraph query failed because Dgraph execution failed because : Sorting not supported on attr: Starship.tags of type: [scalar]" } ], "data": { "queryStarship": [] } } ``` which means we should not allow even list of scalar along with object types in orderable. Only scalar field should be allowed in orderable. So, the correct orderable should be without tags field: ``` enum StarshipOrderable { name length createdAt } ``` This PR fixes the above bug. (cherry picked from commit dc66617) * feat(Dgraph): Add separate compression flag for z and wal dirs (#6401) (#6421) * Add separate compression flag for z and wal dirs * Address comments * Address comments (cherry picked from commit 601cc3b) * docs: Update list documentation (#6455) (#6461) Fixes GRAPHQL-657 Fixes [Discuss Issue](https://discuss.dgraph.io/t/documentation-that-arrays-work-as-a-set-not-storing-duplicates/9590) (cherry picked from commit 07cff6f) * cherry-pick v20.07: fix(GraphQL): Generate correct schema when no orderable field in a ty… (#6460) fix(GraphQL): Generate correct schema when no orderable field in a type (#6456) Fixes GRAPHQL-650 It was introduced after #6392. (cherry picked from commit 8e4d121) * (release/v20.07) test: retry getting client in TestNodes. (#6482) (cherry picked from commit 512c69b) * (release/v20.07) fix(Dgraph): fix bug when deleting and adding to a single UID predicate in the same transaction. (#6449) The iterate method does not account for delete postings in the same transaction so they have to be preserved manually. Otherwise, if a deletion and an add operation to the same uid happen in the same predicate, the deletion operation is not applied. Fixes DGRAPH-1309 (cherry picked from commit 56c8f3c) * Fix(docs): Fix subsection references in docs (#6439) (#6516) * Fix references in docs * Fix mutations/language-rdf-types.md reference (cherry picked from commit 027db53) * Fix(increment): Fix readTs less than minTs (#6317) (#6517) Use the readTs for reads via LocalCache. This would ensure that users don't get a lot of ReadTs less than MinTs errors. This PR also fixes the flaky test in increment_test.go. This has a side-effect on posting list cache. That cache would need to understand versioning to be effective. Co-authored-by: Daniel Mai <daniel@dgraph.io> (cherry picked from commit 2de4675) Co-authored-by: Manish R Jain <manish@dgraph.io> * fix(GraphQL): Fix order and offset in auth queries. (#6221) (#6366) * Fix order and offset in auth queries. * Remove order from top-level auth filter. (cherry picked from commit a5bfd4a) * fix(GraphQL): Fix query rewriting for auth delete when deleting types with inverse field. (#6350) (#6524) * Fix query rewriting for auth delete when deleting types with inverse field. (cherry picked from commit be9ebd0) * cherry-pick v20.07: fix(GraphQL): Link xids properly if there are duplicate xids within t… (#6521) Fixes GRAPHQL-641 See https://discuss.dgraph.io/t/residual-issue-linking-grandchild-in-mutation-with-custom-ids/9613 for the issue details. If the same nested xid was used as a leaf node within an addType mutation, it was only linked the first time that it was used and not subsequently for later objects. This change fixes that by adding the linkage mutations and only removing the mutation which was creating the object with the xid again. (cherry picked from commit 12b2716) # Conflicts: # graphql/e2e/directives/dgraph_directives_test.go # graphql/e2e/normal/normal_test.go # graphql/resolve/add_mutation_test.yaml * fix(GraphQL): fix internal error when doing GraphQL schema introspection after drop all (#6268) (#6525) This PR fixes the "Internal error" response when querying the GraphQL schema after performing drop_all operation. (cherry picked from commit d3bee33) * fix(GraphQL): fix for deletion on interfaces with no non Id field (#6387) (#6417) Fixes GRAPHQL-655. For this user schema (interface have no non-Id field): ``` interface A { name: String! @id } type B implements A { age: Int! } ``` the following delete mutation ``` mutation{ deleteA(filter:{name:{eq: "xyz"}}){ a{ name } } } ``` was resulting in the following error: ``` { "errors": [ { "message": "Internal Server Error - a panic was trapped. This indicates a bug in the GraphQL server. A stack trace was logged. Please let us know by filing an issue with the stack trace." } ] } ``` This PR fixes this bug and now deleting can be performed successfully. (cherry picked from commit 742259b) * fix(GraphQL): fix restoreStatus query with query variables (#6414) (#6424) Fixes GRAPHQL-642. For this restoreStatus query using variable ``` query restoreStatus($restoreId: Int!) { restoreStatus(restoreId: $restoreId) { status errors } } ``` was giving this panic ``` panic: interface conversion: interface {} is json.Number, not int64. ``` Whereas the expected result should be ``` { "data": { "restoreStatus": { "status": "UNKNOWN", "errors": [] } }, "extensions": {} } ``` This PR fixes this panic, Now `resolveStatus` with or without variable works fine. (cherry picked from commit 45afae9) * fix(GraphQl): fix errors from authorization examples given in docs. (#6330) (#6522) This PR fix errors from authorization examples given in docs. (cherry picked from commit 0f3bfb1) * Changes github.com/dgraph-io/graphql-transport-ws version (#6462) (#6529) (cherry picked from commit 5aca255) * Remove auth error from mutation. (#6532) * fix(GraphQL): This PR add schema cleaning in GraphQL. (#6523) * Fix(GraphQL): This PR add schema cleaning in GraphQL. (#6427) Fixes GRAPHQL-572 This PR adds schema cleaning in GraphQL which will remove empty links and types from generated GraphQL schema recursively. (cherry picked from commit bec5322) * fix(GraphQL): Fix squashIntoObject so that results are correctly merged (#6416) (#6530) While creating the add mutation, we weren't squashing different bits correctly which led to the incorrect mutation being sent to Dgraph. This change modifies the set to append for []interface{} to fix that. Fixes GRAPHQL-679 (cherry picked from commit 816a08f) # Conflicts: # graphql/e2e/common/mutation.go # graphql/e2e/directives/schema.graphql # graphql/e2e/normal/schema.graphql # graphql/resolve/schema.graphql * fix(GraphQL): Fix cascade with auth query when RBAC is false (#6444) (#6535) * fix(GraphQL): Fix cascade with auth query when RBAC is false (#6444) * Fix cascade with auth query. * Added additional E2E test. (cherry picked from commit 8c518d4) * fix(GraphQL): Hide info when performing mutation on id field with auth rule. (#6391) (#6534) * fix(GraphQL): Hide info when performing mutation on id field with auth rule. (#6391) * Hide info when performing mutation on id field with auth rule. (cherry picked from commit 5c33428) * fix tests * fix cost test * fix test compilation Co-authored-by: Naman Jain <naman@dgraph.io> Co-authored-by: Apoorv Vardhan <vardhanapoorv@users.noreply.github.com> Co-authored-by: Pawan Rawal <pawan0201@gmail.com> Co-authored-by: Tejas Dinkar <tejas@gja.in> Co-authored-by: Sankalan Parajuli <sankalan@dgraph.io> Co-authored-by: Sankalan Parajuli <sankalan.13@gmail.com> Co-authored-by: Martin Martinez Rivera <martinmr@dgraph.io> Co-authored-by: parasssh <paras@dgraph.io> Co-authored-by: Anurag <anurag@dgraph.io> Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io> Co-authored-by: Martin Martinez Rivera <mrtnz.rvr@gmail.com> Co-authored-by: Tyler <haikusw@gmail.com> Co-authored-by: aman bansal <amanbansal@dgraph.io> Co-authored-by: Harshil Goel <54325286+harshil-goel@users.noreply.github.com> Co-authored-by: Daniel Mai <daniel@dgraph.io> Co-authored-by: vmrajas <rajas@dgraph.io> Co-authored-by: Manish R Jain <manish@dgraph.io> Co-authored-by: Arijit Das <arijit@dgraph.io> Co-authored-by: minhaj-shakeel <minhaj@dgraph.io> Co-authored-by: Jatin Dev <64803093+JatinDevDG@users.noreply.github.com>
abhimanyusinghgaur
added a commit
that referenced
this pull request
Oct 15, 2020
* fix(docs): fix URL paths and make deploy page to not show blank page (#6239) * Fix URL paths (#6189) (cherry picked from commit 4ee4af8) * fix deploy page to not show blank page (cherry picked from commit 89e012b) Co-authored-by: Apoorv Vardhan <vardhanapoorv@users.noreply.github.com> * feat(docs): Port GraphQL docs to v20.07 (#6202) * Port GraphQL docs to v20.07 This reverts commit 8db61a3. Co-authored-by: Pawan Rawal <pawan0201@gmail.com> * doc: Setting the canonical path to each docs without version (#6249) Passing a few variables to the hugo theme * fix: remove unnecessary curly braces from doc (#6256) Co-authored-by: Sankalan Parajuli <sankalan.13@gmail.com> * fix(Dgraph): Stop forcing RAM mode for the write-ahead log. (#6142) (#6259) This change also adds a way to set the table and value log loading modes for the w directory independently of the values for the p directory. Fixes DGRAPH-1898. (cherry picked from commit 5f5aa9c) * Enabling Discuss comments * Fix(Dgraph): Fix how visited nodes are detected in recurse queries. (#6272) (#6276) Currently a node is marked as visited if it's been visited before AND has been visited from the same source UID. In dense graphs, the second condition leads to exponential growth of the data and to other issues such as responses that are too big to encode. Removing this condition fixes the issue. Fixed tests and verified the new output makes sense. Fixes DGRAPH-2337 (cherry picked from commit 3638c12) * fix panicwrap parent check (#6264) (#6299) (cherry picked from commit 52e136f) * release/v20.07 - add --cache_mb and --cache_percentage flag (#6286) * fix(Dgraph): Sort manifests by BackupNum in file handler. (#6263) (#6279) This was being done in the S3 handler but not in the file handler. By default the paths are sorted by the filename but sorting by backup number is more robust. (cherry picked from commit df63d6c) * fix(ACL Query): Fixes queries which use variable at the top level (#6290) (#6298) (cherry picked from commit 6f883a4) * Revert "Fix(Dgraph): Fix how visited nodes are detected in recurse queries. (#6272) (#6276)" (#6306) This reverts commit 9df9f92. * fix(Dgraph): Don't store start_ts in postings. (#6213) Clear out the start_ts field in the postings of deltas before they are marshalled to avoid storing that field in disk. This field is only meant to be used during in-memory processing. Related to https://discuss.dgraph.io/t/start-ts-not-being-cleared-before-postings-are-written-to-disk/9146 (cherry picked from commit fbbd731) * docs: add info about __typename in custom DQL (#6248) (#6273) Reference: [Discuss Post](https://discuss.dgraph.io/t/custom-dql-typename-bug-strange-behaviour/9341) (cherry picked from commit 0faae40) * fix(GraphQL): fixes unexpected fragment behaviour (#6228) (#6274) Fixes: #5516 Fixes: [Discuss Issue](https://discuss.dgraph.io/t/graphql-fragments-generates-unexpected-behaviour/7470) (cherry picked from commit dcce8bf) * fix(task): Return error on closed DB (#6075) (#6320) Fixes DGRAPH-2181 The queries in dgraph are processed in separate goroutines. The pstore badger DB could be closed while the query was being processed. This causes panics such as ``` panic: runtime error: invalid memory address or nil pointer dereference panic: Unclosed iterator at time of Txn.Discard. [signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x11d2311] goroutine 19298 [running]: github.com/dgraph-io/badger/v2.(*Txn).Discard(0xc05586bc20) /go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200718033852-37ee16d8ad1c/txn.go:517 +0xc1 panic(0x19e26a0, 0x2988560) /usr/local/go/src/runtime/panic.go:969 +0x166 github.com/dgraph-io/badger/v2/skl.(*Skiplist).IncrRef(...) ``` This PR attempts to reduce the number of such crashes. This PR doesn't fix the actual issue but it tries to reduce the probability of such crashes by checking if badger is not closed before accessing it. An ideal fix would be to stop all the goroutines started by dgraph while closing DB so that we don't try to read from a closed badger DB. (cherry picked from commit 3cea0fe) * fix(Dgraph): make backups cancel other tasks (#6152) (#6243) Backups should be added to the list of tasks (rollups, snapshots, etc) that are managed by Dgraph so that rollups and other tasks are paused during the backup. (cherry picked from commit ea15b66) * Update badger to v2.2007.2 (#6294) This badger update brings the following changes from badger. https://github.com/dgraph-io/badger/commits/b41e77a Fix(cleanup): Avoid truncating in value.Open on error https://github.com/dgraph-io/badger/commits/a0d4903 fix(cleanup): Do not close cache before compaction * posting list fixes. (#6303) (#6332) (cherry picked from commit deded8b) * fix: Online Restore honors credentials passed in (#6295) (#6302) (cherry picked from commit a8a6e85) Co-authored-by: Tejas Dinkar <tejas@gja.in> * (release/v20.07) Fix(Dgraph): Add a lock to backups to process one request at a time. (#6339) It's possible that two requests reach the server around the same time and send a requests to the alphas with the same backupNum. This could lead to issues further down the line. Related to DGRAPH-2295 (cherry picked from commit 5b79260) * (release/v20.07) fix(Dgraph): Add flags to set table and vlog loading mode for zero. (#6342) Related to DGRAPH-2189 (cherry picked from commit bf79999) * proto fix needed for PR #6331 (#6337) (#6346) (cherry picked from commit 844f34d) * chore(GraphQL): remove skipped tests (#6078) (#6347) Fixes DGRAPH-2357. Fixes GRAPHQL-562. (cherry picked from commit 91e28d5) * test: add --build option for docker-compose up in test script (#6348) (#6352) Fixes DGRAPH-2356 This PR fixes random GraphQL custom logic e2e test failures in TeamCity by requiring docker-compose to always build images before starting the containers, as the mock server in custom logic tests can change any time. (cherry picked from commit 536b317) * cherry-pick v20.07: fix(GraphQL): incorrect generatedSchema in updateGQLSchema (#6349) (#6354) * fix(GraphQL): incorrect generatedSchema in updateGQLSchema (#6349) This PR fixes the behaviour where Dgraph schema was being given as generatedSchema field in updateGQLSchema request. Now, the newly generated complete GraphQL schema is given as generatedSchema. (cherry picked from commit 0b8681c) # Conflicts: # graphql/e2e/schema/schema_test.go * (release/v20.07) fix(Dgraph): add flag to set up compression in zero. (#6355) (cherry picked from commit 254bd29) * fix(dgraph): sentry nil pointer check (#6372) (#6374) (cherry picked from commit 7857fa2) * doc: Adding a note on backend modes * doc: Adding a security page on slash * chore(Docs): Update graph-links.md to remove a duplicate word. (#6378) remove duplicate word "doesn't" from description of two-way edges. * doc: One-Click Documentation for Slash (#6381 #6383) (#6384) (cherry picked from commit e165710) * Use z.Closer instead of y.Closer (#6394) (#6399) The closer type was moved from badger/y to ristretto/z in dgraph-io/ristretto#191 (cherry picked from commit 0287838) * (release/v20.07) docs: Add section on automating backups. (#6405) (cherry picked from commit 0fab710) * relase/v20.07 - Fix(Alpha): MASA: Make Alpha Shutdown Again (#6313) (#6402) * Fix(Alpha): MASA: Make Alpha Shutdown Again (#6313) This PR removes the usage of context.Background() in groups.go and ensures that all the various goroutines exit as intended. Changes: * Shutdown SubscribeForUpdates correctly. * Fix up all the closing conditions. * Consolidate updaters into one closer * Update Badger to master * fix(build): Update ResetAcl args for OSS build. * chore: Remove TODO comment. Co-authored-by: Daniel Mai <daniel@dgraph.io> (cherry picked from commit f1941b3) * fix(shutdown): Force exit if CTRL-C is caught before initialization (#6359) (#6407) Forcefully kill alpha if CTRL-C is caught before node initialization completes. Fixes DGRAPH-2377 Co-authored-by: Manish R Jain <manish@dgraph.io> (cherry picked from commit 41c2052) * Update advanced-queries.md * fix(Query) Fix Star_All delete query when used with ACL enabled (#6331) (#6336) * docs: Add netlify deploy docs to slash docs (#6425) (#6428) (cherry picked from commit d94e5da) * docs: add whitelisting issue error in troubleshoot page to help the community. (#6422) Fixes DGRAPH-2317. Issue is with v20.07 we restrict the admin operations from alpha instances only. But this hasn't been mentioned during the deployment process in any examples. Hence community is running into issues like Unauthorized IP address X.X.X.X. This doc addition in troubleshoot page will help the community members if they run into this issue. * Add badger to docs (#6430) * fix(bulk): Batch list in bulk loader to avoid panic (#6446) * Changelog and Codename (#6488) * fix: Regenerate protos. The different generated proto file makes the release.sh script fail with "Generated protos different in release." Changes * Called "make regenerate" * fix(GraphQL): fix introspection completion bug (#6385) (#6389) Fixes GRAPHQL-673. This PR fixes the issue where introspection queries would break if there were two types implementing same interface and having fields with same name in those two types and that repeating field is also a field in introspection query. This was introduced after #6228. (cherry picked from commit 243a336) * fix(GraphQL): don't generate orderable enum value for list fields (#6392) (#6413) Fixes GRAPHQL-650. For this user schema: ``` type Starship { id: ID! name: String! @search(by: [term]) length: Int64 tags: [String] createdAt: DateTime } ``` we were generating complete GraphQL schema with StarshipOrderable as: ``` enum StarshipOrderable { name length tags createdAt } ``` that would cause a DQL query to be formed which errors out, see this GraphQL query: ``` query { queryStarship(order: {asc: tags}) { id name length tags } } ``` It gives back: ``` { "errors": [ { "message": "Dgraph query failed because Dgraph execution failed because : Sorting not supported on attr: Starship.tags of type: [scalar]" } ], "data": { "queryStarship": [] } } ``` which means we should not allow even list of scalar along with object types in orderable. Only scalar field should be allowed in orderable. So, the correct orderable should be without tags field: ``` enum StarshipOrderable { name length createdAt } ``` This PR fixes the above bug. (cherry picked from commit dc66617) * feat(Dgraph): Add separate compression flag for z and wal dirs (#6401) (#6421) * Add separate compression flag for z and wal dirs * Address comments * Address comments (cherry picked from commit 601cc3b) * docs: Update list documentation (#6455) (#6461) Fixes GRAPHQL-657 Fixes [Discuss Issue](https://discuss.dgraph.io/t/documentation-that-arrays-work-as-a-set-not-storing-duplicates/9590) (cherry picked from commit 07cff6f) * cherry-pick v20.07: fix(GraphQL): Generate correct schema when no orderable field in a ty… (#6460) fix(GraphQL): Generate correct schema when no orderable field in a type (#6456) Fixes GRAPHQL-650 It was introduced after #6392. (cherry picked from commit 8e4d121) * (release/v20.07) test: retry getting client in TestNodes. (#6482) (cherry picked from commit 512c69b) * (release/v20.07) fix(Dgraph): fix bug when deleting and adding to a single UID predicate in the same transaction. (#6449) The iterate method does not account for delete postings in the same transaction so they have to be preserved manually. Otherwise, if a deletion and an add operation to the same uid happen in the same predicate, the deletion operation is not applied. Fixes DGRAPH-1309 (cherry picked from commit 56c8f3c) * Fix(docs): Fix subsection references in docs (#6439) (#6516) * Fix references in docs * Fix mutations/language-rdf-types.md reference (cherry picked from commit 027db53) * Fix(increment): Fix readTs less than minTs (#6317) (#6517) Use the readTs for reads via LocalCache. This would ensure that users don't get a lot of ReadTs less than MinTs errors. This PR also fixes the flaky test in increment_test.go. This has a side-effect on posting list cache. That cache would need to understand versioning to be effective. Co-authored-by: Daniel Mai <daniel@dgraph.io> (cherry picked from commit 2de4675) Co-authored-by: Manish R Jain <manish@dgraph.io> * fix(GraphQL): Fix order and offset in auth queries. (#6221) (#6366) * Fix order and offset in auth queries. * Remove order from top-level auth filter. (cherry picked from commit a5bfd4a) * fix(GraphQL): Fix query rewriting for auth delete when deleting types with inverse field. (#6350) (#6524) * Fix query rewriting for auth delete when deleting types with inverse field. (cherry picked from commit be9ebd0) * cherry-pick v20.07: fix(GraphQL): Link xids properly if there are duplicate xids within t… (#6521) Fixes GRAPHQL-641 See https://discuss.dgraph.io/t/residual-issue-linking-grandchild-in-mutation-with-custom-ids/9613 for the issue details. If the same nested xid was used as a leaf node within an addType mutation, it was only linked the first time that it was used and not subsequently for later objects. This change fixes that by adding the linkage mutations and only removing the mutation which was creating the object with the xid again. (cherry picked from commit 12b2716) # Conflicts: # graphql/e2e/directives/dgraph_directives_test.go # graphql/e2e/normal/normal_test.go # graphql/resolve/add_mutation_test.yaml * fix(GraphQL): fix internal error when doing GraphQL schema introspection after drop all (#6268) (#6525) This PR fixes the "Internal error" response when querying the GraphQL schema after performing drop_all operation. (cherry picked from commit d3bee33) * fix(GraphQL): fix for deletion on interfaces with no non Id field (#6387) (#6417) Fixes GRAPHQL-655. For this user schema (interface have no non-Id field): ``` interface A { name: String! @id } type B implements A { age: Int! } ``` the following delete mutation ``` mutation{ deleteA(filter:{name:{eq: "xyz"}}){ a{ name } } } ``` was resulting in the following error: ``` { "errors": [ { "message": "Internal Server Error - a panic was trapped. This indicates a bug in the GraphQL server. A stack trace was logged. Please let us know by filing an issue with the stack trace." } ] } ``` This PR fixes this bug and now deleting can be performed successfully. (cherry picked from commit 742259b) * fix(GraphQL): fix restoreStatus query with query variables (#6414) (#6424) Fixes GRAPHQL-642. For this restoreStatus query using variable ``` query restoreStatus($restoreId: Int!) { restoreStatus(restoreId: $restoreId) { status errors } } ``` was giving this panic ``` panic: interface conversion: interface {} is json.Number, not int64. ``` Whereas the expected result should be ``` { "data": { "restoreStatus": { "status": "UNKNOWN", "errors": [] } }, "extensions": {} } ``` This PR fixes this panic, Now `resolveStatus` with or without variable works fine. (cherry picked from commit 45afae9) * fix(GraphQl): fix errors from authorization examples given in docs. (#6330) (#6522) This PR fix errors from authorization examples given in docs. (cherry picked from commit 0f3bfb1) * Changes github.com/dgraph-io/graphql-transport-ws version (#6462) (#6529) (cherry picked from commit 5aca255) * Remove auth error from mutation. (#6532) * fix(GraphQL): This PR add schema cleaning in GraphQL. (#6523) * Fix(GraphQL): This PR add schema cleaning in GraphQL. (#6427) Fixes GRAPHQL-572 This PR adds schema cleaning in GraphQL which will remove empty links and types from generated GraphQL schema recursively. (cherry picked from commit bec5322) * fix(GraphQL): Fix squashIntoObject so that results are correctly merged (#6416) (#6530) While creating the add mutation, we weren't squashing different bits correctly which led to the incorrect mutation being sent to Dgraph. This change modifies the set to append for []interface{} to fix that. Fixes GRAPHQL-679 (cherry picked from commit 816a08f) # Conflicts: # graphql/e2e/common/mutation.go # graphql/e2e/directives/schema.graphql # graphql/e2e/normal/schema.graphql # graphql/resolve/schema.graphql * fix(GraphQL): Fix cascade with auth query when RBAC is false (#6444) (#6535) * fix(GraphQL): Fix cascade with auth query when RBAC is false (#6444) * Fix cascade with auth query. * Added additional E2E test. (cherry picked from commit 8c518d4) * fix(GraphQL): Hide info when performing mutation on id field with auth rule. (#6391) (#6534) * fix(GraphQL): Hide info when performing mutation on id field with auth rule. (#6391) * Hide info when performing mutation on id field with auth rule. (cherry picked from commit 5c33428) * fix(GraphQL): This PR adds documentation for Scalar DateTime type. (#6373) (#6554) Fixes GRAPHQL-631 This PR adds documentation for Scalar DateTime type. (cherry picked from commit 91c81ca) * Revert "fix(GraphQL): This PR add schema cleaning in GraphQL. (#6523)" (#6559) This reverts commit 812a6f0. # Conflicts: # graphql/schema/testdata/schemagen/output/filter-cleanSchema-all-empty.graphql # graphql/schema/testdata/schemagen/output/filter-cleanSchema-circular.graphql # graphql/schema/testdata/schemagen/output/filter-cleanSchema-custom-mutation.graphql # graphql/schema/testdata/schemagen/output/filter-cleanSchema-directLink.graphql * Update index.md (#6566) (#6567) (cherry picked from commit 512c0a9) * perf(raft): Use raft storage in managedmode (#6457) (#6547) This PR changes how we use badger in the wal store. Currently, we run badger in normal mode for `w` and `zw` store. We've seen up to 900K entries for the same hard state (hs) key in `zw` store. These duplicate keys cause spikes in read latencies in `zw`. The `w` store has more compactions compared to `zw` store and so it has lesser stale data and thus lesser spikes in read latencies. The fix here is to open the `w` and `zw` directories in managed mode and perform all writes on the same timestamp (max version in the db). This leads to close to 0 duplicates in the store. This PR also fixes the raft leader election issue which is a result of high read latencies. (cherry picked from commit 6882e37) * chore(query): Return content length header for queries (#6469) (#6480) This PR adds `Content-Length` header to the response of queries sent via HTTP. Fixes DGRAPH-1675 (cherry picked from commit ed63786) * fix(bulkLoader): Use flags for cache (#6322) (#6467) Bulk loader uses caches in compression and this PR adds flags to make it configurable. Bulk loader was setting compressionLevel but not the compression option. As a result of this, badger wasn't compressing any data. This PR fixes this setting compression if compressionLevel is greater than 0. (cherry picked from commit 99341dc) * docs: add documentation for clone backend (#6622) * (release/v20.07): feat(Dgraph): add utility to export backup data. (#6590) * (release/v20.07): feat(Dgraph): add utility to export backup data. (#6550) This utility allows to take a single backup (full or incremental) and export the data and the schema inside it to RDF. Encrypted backups are supported. Fixes DGRAPH-2465 (cherry picked from commit 369a5c1) * Remove destination field. * (release/v20.07) build: Makefile to build Dgraph inside docker container. (#6601) The script is useful for situations in which the host system creates a binary that's incompatible with the docker container (e.g different versions of glibc). (cherry picked from commit c789738) * (release/v20.07) Pause rollups during snapshot streaming. (#6611) (cherry picked from commit b777913) * (release/v20.07) test: Fix flakiness in group_delete_test.go (#6626) The test and utility were assuming that node 1 was in group 1, node 2 in group 2, and so on. This is not true all the time, hence the flakiness. Fixing the code to not make this assumptions as well as retrying the DropAll in NodesSetup to ensure all the nodes are up before proceeding with the test. (cherry picked from commit 36638a7) * CID never created if Zero stops early after first init (#5719) (#6637) (cherry picked from commit 09ef988) * test: Enterprise License tests (#5822) (#6640) Enterprise License unit tests * Pagination param "after" does not work when using func: uid(v) (#6365) (#6634) fixed : Pagination param "after" does not work when using func: uid(v) * (release/v20.07) Update nextRaftId when starting a node with a raftId > 0. (#6597) (cherry picked from commit a4f21f4) * release/v20.07 - fix(gqlgen): Update gqlgen in go.mod (#6594) (#6646) * fix(gqlgen): Update gqlgen in go.mod (#6594) * Add tests for querying alias and attribute on interface * Requested changes (cherry picked from commit f19d882) * Fix test * Fix(GraphQL): fix object Linking with `hasInverse` (#6557) (#6648) This PR fixes incorrect linking of objects with `hasInverse`. For example For this schema, `type Country` has field `state` with `hasInverse` predicate. ``` type Country { id: ID! name: String! @search(by: [trigram, hash]) states: [State] @hasInverse(field: country) } type State { id: ID! xcode: String! @id @search(by: [regexp]) name: String! capital: String country: Country } ``` And for the following mutation ``` mutation addCountry($input: [AddCountryInput!]!) { addCountry(input: $input) { country { name states{ xcode name country{ name } } } } } ``` with `input`: ``` { "input": { "name": "A Country", "states": [ { "xcode": "abc", "name": "Alphabet" }, { "xcode": "def", "name": "Vowel", "country": { "name": "B country" } } ] } } ``` should result in ``` { "addCountry": { "country": [ { "name": "A country", "states": [ { "country": { "name": "A country" }, "name": "Alphabet", "xcode": "abc" }, { "country": { "name": "A country" }, "name": "Vowel", "xcode": "def" } ] } ] } }` ``` whereas the incorrect output was ``` { "addCountry": { "country": [ { "name": "A country", "states": [ { "country": { "name": "A country" }, "name": "Alphabet", "xcode": "abc" }, { "country": { "name": "B country" }, "name": "Vowel", "xcode": "def" } ] } ] } }` ``` This PR fixes this issue. (cherry picked from commit e496467) * doc: update /admin endpoint documentation. (#6415) (#6527) This PR updates documentation for the `/admin` endpoint operation and `restore` operation. (cherry picked from commit 288eafb) * fix(GraphQL Query): Remove extra fields when querying interfaces (#6596) (#6647) (cherry picked from commit cec5567) This PR fixes: GRAPHQL-638 * fix (graphql): disallowing field names with as (#6645) This is related to GRAPHQL-564. as is Dgraph reserved keyword. It's being used by DQL to identify variables. This PR is to restrict the naming for fields with the name as. * (release/v20.07) docs: Update restore documentation. (#6653) (cherry picked from commit 014ee10) * fix(Query): Fix wrong path response for k-shortest paths (#6437) (#6654) The k-shortest path query sometimes returns a wrong response. The cause of this issue is inappropriate use of sync.pools. Whenever a valid path is found a new variable is assigned with this route and that variable is appended in the kroutes (which is then parsed to produce response). But the route is also put in the sync.pool. Since the route is a structure containing a pointer to slice along with other fields, when it is fetched back from the pool in future then any mutation on this fetched route causes modification in the kroutes, resulting in garbage response. This issue is fixed by making a copy of the route struct rather than just assigning it to a variable, which causes a shallow copy. (cherry picked from commit 4792d8b) * Updater badger to 5e3d4b9 (#6669) * Break out if g.Ctx is done (#6675) (cherry picked from commit a8a47af) Co-authored-by: Manish R Jain <manish@dgraph.io> * fix(Auth): fix Poor-man's auth for admin operations (#6660) (#6686) Fixes DGRAPH-2419 Fixes [Discuss Issue](https://discuss.dgraph.io/t/acl-login-will-fail-if-auth-token-enabled-in-v20-07-0/10044) This PR fixes Poor-man's auth for following endpoints: * `/login` * `/admin` (cherry picked from commit 4fc328d) # Conflicts: # graphql/e2e/common/common.go * Merge branch 'release/v20.07' of github.com:dgraph-io/dgraph into abhimanyu/release-slash-merge # Conflicts: # dgraph/cmd/alpha/run.go # dgraph/cmd/bulk/reduce.go # dgraph/cmd/zero/raft.go # edgraph/server.go # go.mod # go.sum # graphql/e2e/common/common.go # graphql/e2e/common/fragment.go # graphql/e2e/common/mutation.go # graphql/e2e/common/query.go # graphql/e2e/custom_logic/custom_logic_test.go # graphql/e2e/directives/schema_response.json # graphql/e2e/normal/schema_response.json # graphql/e2e/schema/generatedSchema.graphql # graphql/resolve/resolver.go # graphql/schema/testdata/schemagen/output/type-without-orderables.graphql # protos/pb/pb.pb.go # wiki/content/enterprise-features/binary-backups.md # wiki/content/graphql/admin/index.md # wiki/content/graphql/schema/reserved.md * fix tests Co-authored-by: Naman Jain <naman@dgraph.io> Co-authored-by: Apoorv Vardhan <vardhanapoorv@users.noreply.github.com> Co-authored-by: Pawan Rawal <pawan0201@gmail.com> Co-authored-by: Tejas Dinkar <tejas@gja.in> Co-authored-by: Sankalan Parajuli <sankalan@dgraph.io> Co-authored-by: Sankalan Parajuli <sankalan.13@gmail.com> Co-authored-by: Martin Martinez Rivera <martinmr@dgraph.io> Co-authored-by: parasssh <paras@dgraph.io> Co-authored-by: Anurag <anurag@dgraph.io> Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io> Co-authored-by: Martin Martinez Rivera <mrtnz.rvr@gmail.com> Co-authored-by: Tyler <haikusw@gmail.com> Co-authored-by: aman bansal <amanbansal@dgraph.io> Co-authored-by: Harshil Goel <54325286+harshil-goel@users.noreply.github.com> Co-authored-by: Daniel Mai <daniel@dgraph.io> Co-authored-by: vmrajas <rajas@dgraph.io> Co-authored-by: Manish R Jain <manish@dgraph.io> Co-authored-by: Arijit Das <arijit@dgraph.io> Co-authored-by: minhaj-shakeel <minhaj@dgraph.io> Co-authored-by: Jatin Dev <64803093+JatinDevDG@users.noreply.github.com> Co-authored-by: Akash Jain <akashjain971@gmail.com> Co-authored-by: Neeraj Battan <66044626+antblood@users.noreply.github.com> Co-authored-by: aman bansal <bansalaman2905@gmail.com> Co-authored-by: Ahsan Barkati <ahsanbarkati@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This would ensure that users don't get a lot of ReadTs less than MinTs errors. This PR also fixes the flaky test in increment_test.go.
This has a side-effect on posting list cache. That cache would need to understand versioning to be effective.
This change is