merging upstream #1
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR fixes an error returned if there are no uids for geo query.
{
tourist(func: near(loc, [-122.469829, 37.771935], 1000) ) {
name
}
}
It used to return an error while it should return an empty result.
Bumps [pyyaml](https://github.com/yaml/pyyaml) from 5.3.1 to 5.4. - [Release notes](https://github.com/yaml/pyyaml/releases) - [Changelog](https://github.com/yaml/pyyaml/blob/master/CHANGES) - [Commits](yaml/pyyaml@5.3.1...5.4) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
) Bumps [pyyaml](https://github.com/yaml/pyyaml) from 5.3.1 to 5.4. - [Release notes](https://github.com/yaml/pyyaml/releases) - [Changelog](https://github.com/yaml/pyyaml/blob/master/CHANGES) - [Commits](yaml/pyyaml@5.3.1...5.4) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Bumps [rsa](https://github.com/sybrenstuvel/python-rsa) from 3.4.2 to 4.1. - [Release notes](https://github.com/sybrenstuvel/python-rsa/releases) - [Changelog](https://github.com/sybrenstuvel/python-rsa/blob/main/CHANGELOG.md) - [Commits](sybrenstuvel/python-rsa@version-3.4.2...version-4.1) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* update VAULT_VERSION=latest for vault docker image. * add instructions for using bound_cidr_list role where sercret-id is no longer needed.
This commit implements map-reduce based online restore. The offline restore has been removed. It also implements backup-export on top of map-reduce based restore. Co-authored-by: Manish R Jain <manish@dgraph.io>
…gn (#7666) This PR along with the previous restore PR is a BREAKING change. Marking this PR as breaking, because we forgot to mark the previous one. - Make restore map run concurrently for faster execution. - Add progress updates every second for both map and reduce phase. - Refactor code to break out the map and reduce code into separate files. - Make reduce cheap by avoiding marshal-unmarshal step. With these changes, I see map phase is faster than reduce and both finish in about 2 mins each. Map runs at 200 MBps, while Reduce runs at 130 MBps, processing 20GB of uncompressed data in under 5 mins. Changes: * Work on optimizing restore * Some file moves * Moved map output to a temp directory. * Add restoreTs in export-backup Co-authored-by: Ahsan Barkati <ahsanbarkati@gmail.com>
…t levels with same alias are selected. (#7639) We are generating incorrect order of the fields in a result of the @normalize directive when there are multiple fields with same Alias at nested levels. As a result of that in resulting JSON, the nodes with the same Alias were not converted to list. For example, if we have below data in dgraph . { "_id": "Process", "Name": "Process", "Extends": { "_id": "Core/Namespace", "Name": "Namespace" } } And if we do below query with @recurse and @normalize directive { item as var(func:eq(_id,"Process")) b2(func:uid(item))@normalize@recurse { Name:Name id:_id Extends } } We will get this response from Dgraph where fields with same alias are not together and while converting response to JSON , we are not able to combine fields with same Alias in a list. "b2": [ { "Name": "Process", "id": "Process", "Name":"Namespace", "id": "Core/Namespace" } ] In this PR we fix this in normalize, by putting all fields with the same Alias together. So now response of above query will be "b2": [ { "Name": [ "Process", "Namespace" ], "id": [ "Process", "Core/Namespace" ] } ] This behaviour is broken by PR #6362. Prior to this , we used sorting on fields which guarantee that fields with same name or Alias comes together.
Append the directory prefix in the s3 paths for rename operation.
… (GRAPHQL-1118) (#6649) This PR makes Zero HTTP endpoints available in GraphQL admin. It also adds a flag `--disable_admin_http` in Zero which can be used to disable the administrative HTTP endpoints in Zero. The following are considered as the administrative endpoints for Zero: * `/state` * `/removeNode` * `/moveTablet` * `/assign` * `/enterpriseLicense` RFC: https://discuss.dgraph.io/t/moving-zero-http-endpoints-to-alpha-graphql-admin/10725
…7665) Bumps [pygments](https://github.com/pygments/pygments) from 2.6.1 to 2.7.4. - [Release notes](https://github.com/pygments/pygments/releases) - [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES) - [Commits](pygments/pygments@2.6.1...2.7.4) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* update compose/run.sh to work on mac, support new bin path, modularize script for readability
- Gzip compression is slow. For backups, I've switched to using snappy for compression. This change is backwards compatible. - Refactor how backup reader is generated to deal with the compression that's used for creating backup. - Avoid creating a z.Buffer for every call to marshal uid list. Instead, reuse the same one over and over. With these changes, we can backup 150GB of data in ~50 mins on my workstation, using 8 goroutines and under 1.5GB of jemalloc memory.
With the refactor of File System and Minio handlers in enterprise backup code, the operations that they now do are pretty standard. So, we can move that code out to x package, under open source, to be used by export as well.
This PR adds support for language tags in GraphQL. This feature is available in Dgraph. We exposed language tags to GraphQL using @dgraph directive. RFC: https://discuss.dgraph.io/t/rfc-allow-language-tag-support-in-graphql/13027 Co-authored-by: Abhimanyu Singh Gaur <12651351+abhimanyusinghgaur@users.noreply.github.com>
* Remove unwanted arrays * Simplify code and add comments * Fix lint errors
* Remove unused function * Add tracing for existence query execution * Fix tests
) There might be a scenario where in a multi-tenant cluster a malicious user leases out a large number of UIDs or bump it to a large number, causing an issue for the other tenants. This PR adds a super flag --limit to zero. This flag limits the number of UIDs (uid-lease=100000;) that can be leased by a namespace within an interval specified by refill-interval=30s;. This PR also changes the xidmap's New() API as we need to pass in the dgo client. Along with it, it does some minor cleaning where flag parsing was in the critical path.
…#7632) The GetNew function may fetch the posting list from the cache. This cached copy of the posting list might be still in use at other places and so we need to acquire a lock on it before we read it. Co-authored-by: NamanJain8 <jnaman806@gmail.com>
Fix queries involving `regexp`, `allofterms`, `alloftext` and `match` function with pagination. These functions relies on indexes and needs to fetch postings/uids for each relevant index key to generate the final result. Remove early pagination for these cases.
There was some noise in logs due to errors printed when JWT was not found in context. This creates a bad user experience and creates confusion for users. This PR just don't print these error logs. The proper handling is done in the downstream functions.
* Return error if operand for math functions are invalid
For the upgrade in versions <= 20.07, dgraph upgrade tool updates the acl relates data. Hence, it had --acl flag. With the update of the upgrade tool for 21.03, it should work without ACL as well. This PR adds that support.
When there was a single plist to start from and there was nothing in mutationMap, the encode does not happen and hence the splits were not getting generated. This PR fixes that issue.
Open-source the backup and restore feature.
…isition (#8068) Optimize populateSchema() by avoiding repeated lock acquisition. we can get the schema for the predicate once and then check for the required field without taking a read lock.
The bulk loader creates the p directory and hence should have the correct external magic version. With the change that introduced the magic version, somehow bulk loader got missed out.
Negative offsets (e.g., offset: -4) can cause panics when sorting. This can happen when the query has the following characteristics:
1. The query is sorting on an indexed predicate
2. The results include nodes that also don't have the sorted predicate
3. A negative offset is used.
(panic trace is from v20.11.2-rc1-23-gaf5030a5)
panic: runtime error: slice bounds out of range [-4:]
goroutine 1762633 [running]:
github.com/dgraph-io/dgraph/worker.sortWithIndex(0x1fb12e0, 0xc00906a880, 0xc009068660, 0x0)
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/sort.go:330 +0x244d
github.com/dgraph-io/dgraph/worker.processSort.func2(0x1fb12e0, 0xc00906a880, 0xc009068660, 0xc0090686c0)
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/sort.go:515 +0x3f
created by github.com/dgraph-io/dgraph/worker.processSort
/ext-go/1/src/github.com/dgraph-io/dgraph/worker/sort.go:514 +0x52a
The Docker SDK will not allow mapping a port that it not explicitly exposed in Dockerfile (the docker cli does not have the same restrictions, any port can be mapped with the -P command line parameter). For more background: https://discuss.dgraph.io/t/docker-compose-dgraph-standalone/14635
Fixes 2 race conditions: - Txn's MaxAssignedSeen and AppliedIndexSeen were accessed without locks, this leads to a read-write race. - Rollups are not supposed to modify the List.plist. But it was modifying the plist by assigning it in out.plist that further modifies it by running splits.
Update ACL tests to cover the behavior for dgraph.all permissions with reverse edges. When the dgraph.all permission is set, it also applies to reverse edges.
* chore(release): adding changelog for v21.03 (#7696) * adding changelog for v21.03 (cherry picked from commit a77bbe8) * update changelog (#7905) Update the changelog. (cherry picked from commit ea1cb5f) * doc(changelog): Add v21.03.2 changelog. (#8000) (cherry picked from commit a3d41b1) Co-authored-by: aman bansal <bansalaman2905@gmail.com> Co-authored-by: Ahsan Barkati <ahsan@dgraph.io> Co-authored-by: Daniel Mai <daniel@dgraph.io>
…#8092) This PR improves the performance of the rollups. Also, it fixes memory issues of the bulk loader. - Use the optimized Split API from the Sroar to split the bitmap (while doing rollup). This is an improvement over the recursive binary split that was very slow. - In bulk loader, use HandoverSkipList API instead of WriteBatch to avoid the memory constraints of doing a huge batch write. - Also, it introduces BitForbidPosting which limits the capability to store large posting lists that generate splits of length greater than the --limit max-splits flag. Once a posting list is marked as Forbidden it cannot be recovered.
… the tablets (#8… (#8100) * feat: adding bulk call for alpha to inform zero about the tablets (#8089) * adding bulk call for alpha to inform zero about the tablets (#8088) (cherry picked from commit c204d0a) * fix(zero): fix update membership to make bulk tablet proposal instead of multiple small (#8090) Converting the smaller tablet proposals into a single bulk call makes the execution faster. Else, this update becomes of the order of O(n^2), where n is the number of tablet updates. (cherry picked from commit 695c6d7) Co-authored-by: aman bansal <bansalaman2905@gmail.com>
…8101) There is an issue in ExtractNamespaceFromPredicate. The issue is the parsing was done assuming ns in <ns>-<attr> to be decimal (actually it is hexadecimal). This leads to the following issues. A predicate a-name, it was skipped. A predicate 11-name was parsed as namespace 11, actually it is namespace 17 (0x11). (cherry picked from commit 0c9f601)
(cherry picked from commit 89b573a)
* chore(changelog): adding changelog for 21.12 * adding tag name
* Go 1.17 * Node v14
"Dgraph version" should now be based on v21.12.
* opening up whitelist for local setup * update to private address cidrs * increase lint test timeout * Revert "increase lint test timeout" This reverts commit 5f2ded8. * Update golangci-lint.yml Fix OOM errors using GOGC Co-authored-by: Akon Dey <akon-dey@users.noreply.github.com>
* [draft] fix(ci): ci porting * adding more build stages adding next build stages * proto & unit tests adds unit tests & Proto tests * protobuf compiler adds protobuf compiler step * Update go.yml * debug commit env specific debug commit * go path go path issues * Update go.yml * test fixes test fixes * testing testing * Update and rename go.yml to CI.yml finalized all steps * port PR #8144 * fix gitignore for test logs * fix hmac, dummy secrets were pushed * rm dummy secrets * Update golangci-lint.yml increase timeouts
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
30 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.