Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test: Enterprise License tests #5822

Merged
merged 29 commits into from Jul 16, 2020
Merged

test: Enterprise License tests #5822

merged 29 commits into from Jul 16, 2020

Conversation

antblood
Copy link
Contributor

@antblood antblood commented Jul 6, 2020

Adding files for Enterprise license unit testing and Integration testing.
Fixes DGRAPH-1348
There are three tests in it:

  1. Applying expired license, this will get applied but it should not be enabled and user information should be extracted
  2. Applying invalid license, this won't get applied
  3. Applying empty license, this also won't get applied

This change is Reviewable

Docs Preview: Dgraph Preview

@antblood antblood changed the title Neeraj/license testing Enterprise License unit testing Jul 6, 2020
@antblood antblood changed the title Enterprise License unit testing #WIP Enterprise License unit testing Jul 6, 2020
Copy link
Contributor

@pawanrawal pawanrawal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 4 of 4 files at r1.
Reviewable status: all files reviewed, 6 unresolved discussions (waiting on @antblood, @manishrjain, and @vvbalaji-dgraph)


dgraph/cmd/zero/enterpriseExperiedKey.key, line 13 at r1 (raw file):

	DQTievpyQF47HzuifvqUyxDSEsYiSGhhap1e/tvf1VaZoFUuTYQQpiV7+9K3UrL0
	SnJ5TRWS7cEKBLyZszrakGpqVakvEUlWO4wG0Fp4kUa4YXw8t58oqHRn9aAtoqJd
	UOLnq2semUttaySR4DHhjneO3/RoVm79/aaqMi/QNJzc9Tt9nY0AgcYlA3bVXmAZ

ExpiredKey and not Experied


dgraph/cmd/zero/enterpriseUnit_test.go, line 22 at r1 (raw file):

	}{
		{
			"Using expired entrerprised license key should return an error",

enterprise


dgraph/cmd/zero/enterpriseUnit_test.go, line 24 at r1 (raw file):

			"Using expired entrerprised license key should return an error",
			expiredKey,
			true,

Is this ever false? If not then do we even need it?


dgraph/cmd/zero/enterpriseUnit_test.go, line 41 at r1 (raw file):

	for _, tt := range tests {
		t.Logf("Running: %s\n", tt.name)
		err := server.applyLicenseFileReturnError(expiredKey)

Shouldn't you be taking this value from tt.licenseKeyPath?


dgraph/cmd/zero/license_ee.go, line 151 at r1 (raw file):

}

func (s *Server) applyLicenseFileReturnError(path string) error {

Change the name to applyLicenseFile. No need to add ReturnError because that is clear from the function definition itself.


dgraph/cmd/zero/license_ee.go, line 154 at r1 (raw file):

	content, err := ioutil.ReadFile(path)
	if err != nil {
		glog.Infof("Unable to apply license at %v due to error %v", path, err)

Do we have a test for this as well i.e. for a non-existent file? Why are we printing the error if we are already returning it to the user as well?

@antblood antblood changed the title #WIP Enterprise License unit testing Enterprise License unit testing Jul 14, 2020
Copy link
Contributor

@harshil-goel harshil-goel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 2 of 3 files reviewed, 13 unresolved discussions (waiting on @antblood, @manishrjain, and @vvbalaji-dgraph)


dgraph/cmd/zero/enterpriseExperiedKey.key, line 2 at r2 (raw file):

-----BEGIN PGP MESSAGE-----
	owEBwgI9/ZANAwAKAXomeeH3SyppAax7YgxsaWNlbnNlLmpzb25etF5+ewogICJ1

Indentation problem


dgraph/cmd/zero/enterpriseIntegration_test.go, line 20 at r2 (raw file):

)

var expiredKey = []byte(`-----BEGIN PGP MESSAGE-----

Read it from the file itself.


dgraph/cmd/zero/enterpriseIntegration_test.go, line 47 at r2 (raw file):

	var (
		dgraph = flag.String("d", "127.0.0.1:9080", "Dgraph Alpha address")

check testutil.SockAddr, we already have a dgraph alpha address as a input, use that instead.


dgraph/cmd/zero/enterpriseIntegration_test.go, line 70 at r2 (raw file):

		log.Fatal(err)
	}
	fmt.Printf("Response: %s\n", resp.Json)

Remove this


dgraph/cmd/zero/enterpriseIntegration_test.go, line 72 at r2 (raw file):

	fmt.Printf("Response: %s\n", resp.Json)

	enterpriseLicenseURL := "http://localhost:6080/enterpriseLicense"

What is at 6080?


dgraph/cmd/zero/enterpriseIntegration_test.go, line 100 at r2 (raw file):

		require.NoError(t, err)
		correctError := strings.Contains(string(responseBody), tt.expectedOutput)
		// errors := unmarshaledBody.(map[string]interface{})["errors"].([]interface{})[0].(map[string]interface{})["message"]

Remove this


dgraph/cmd/zero/enterpriseInvalidKey.key, line 2 at r2 (raw file):

-----BEGIN PGP MESSAGE-----
	owEBwgI9/ZANAwAKAXomeeH3SyppAax7YgxsaWNlbnNlLmpzb25etF5+ewogICJ1

Indentation error

Copy link
Contributor Author

@antblood antblood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 2 of 3 files reviewed, 13 unresolved discussions (waiting on @antblood, @harshil-goel, @manishrjain, @pawanrawal, and @vvbalaji-dgraph)


dgraph/cmd/zero/enterpriseIntegration_test.go, line 47 at r2 (raw file):

Previously, harshil-goel (Harshil Goel) wrote…

check testutil.SockAddr, we already have a dgraph alpha address as a input, use that instead.

Done.


dgraph/cmd/zero/enterpriseIntegration_test.go, line 70 at r2 (raw file):

Previously, harshil-goel (Harshil Goel) wrote…

Remove this

Done.


dgraph/cmd/zero/enterpriseIntegration_test.go, line 72 at r2 (raw file):

Previously, harshil-goel (Harshil Goel) wrote…

What is at 6080?

Done.


dgraph/cmd/zero/enterpriseIntegration_test.go, line 100 at r2 (raw file):

Previously, harshil-goel (Harshil Goel) wrote…

Remove this

Done.


dgraph/cmd/zero/enterpriseInvalidKey.key, line 2 at r2 (raw file):

Previously, harshil-goel (Harshil Goel) wrote…

Indentation error

Done.


dgraph/cmd/zero/enterpriseUnit_test.go, line 22 at r1 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

enterprise

Done.


dgraph/cmd/zero/enterpriseUnit_test.go, line 24 at r1 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

Is this ever false? If not then do we even need it?

Done.


dgraph/cmd/zero/enterpriseUnit_test.go, line 41 at r1 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

Shouldn't you be taking this value from tt.licenseKeyPath?

Done.


dgraph/cmd/zero/license_ee.go, line 151 at r1 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

Change the name to applyLicenseFile. No need to add ReturnError because that is clear from the function definition itself.

Done.


dgraph/cmd/zero/license_ee.go, line 154 at r1 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

Do we have a test for this as well i.e. for a non-existent file? Why are we printing the error if we are already returning it to the user as well?

Done.

Copy link
Contributor Author

@antblood antblood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 2 of 3 files reviewed, 13 unresolved discussions (waiting on @harshil-goel, @manishrjain, @pawanrawal, and @vvbalaji-dgraph)


dgraph/cmd/zero/enterpriseExperiedKey.key, line 13 at r1 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

ExpiredKey and not Experied

Done.


dgraph/cmd/zero/enterpriseExperiedKey.key, line 2 at r2 (raw file):

Previously, harshil-goel (Harshil Goel) wrote…

Indentation problem

Done.


dgraph/cmd/zero/enterpriseIntegration_test.go, line 20 at r2 (raw file):

Previously, harshil-goel (Harshil Goel) wrote…

Read it from the file itself.

Done.

Copy link
Contributor

@martinmr martinmr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the future, use "git rm" and "git mv" when removing and moving files so that they are tracked appropriately. Otherwise the changelist looks off.

One small comment but otherwise it :lgtm:

Reviewed 1 of 3 files at r3.
Reviewable status: all files reviewed, 15 unresolved discussions (waiting on @antblood, @harshil-goel, @manishrjain, @pawanrawal, and @vvbalaji-dgraph)


dgraph/cmd/zero/enterpriseIntegration_test.go, line 1 at r3 (raw file):

package zero

license_test.go should be enough.


dgraph/cmd/zero/enterpriseIntegration_test.go, line 67 at r3 (raw file):

		}
		require.NoError(t, err)
		require.Equal(t, true, correctError)

I think there's a require.Contains function you can use here to directly compare the two strings.

Copy link
Contributor

@martinmr martinmr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Marking as approved.

Reviewable status: all files reviewed, 15 unresolved discussions (waiting on @antblood, @harshil-goel, @manishrjain, @pawanrawal, and @vvbalaji-dgraph)

@martinmr martinmr changed the title Enterprise License unit testing test: Enterprise License unit testing Jul 15, 2020
@martinmr martinmr changed the title test: Enterprise License unit testing test: Enterprise License tests Jul 15, 2020
@antblood antblood closed this Jul 16, 2020
@antblood antblood deleted the neeraj/LicenseTesting branch July 16, 2020 05:46
Copy link
Contributor

@pawanrawal pawanrawal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 2 files reviewed, 19 unresolved discussions (waiting on @antblood, @harshil-goel, @manishrjain, @martinmr, @pawanrawal, and @vvbalaji-dgraph)


systest/license/license_test.go, line 71 at r4 (raw file):

		name           string
		licenseKey     []byte
		expectError    bool

not needed if its false always


systest/license/license_test.go, line 74 at r4 (raw file):

		code           string
		user           string
		licenseEnabled interface{}

not needed if its nil always


systest/license/license_test.go, line 110 at r4 (raw file):

		responseBody, err := ioutil.ReadAll(response.Body)
		require.NoError(t, err)
		json.Unmarshal(responseBody, &enterpriseResponse)

We should also compare the error message returned to the user. Also, check errors that are returned from json.Unmarshal


systest/license/license_test.go, line 124 at r4 (raw file):

			responseBody, err := ioutil.ReadAll(response.Body)
			require.NoError(t, err)
			json.Unmarshal(responseBody, &stateResponse)

check error here

Copy link
Contributor Author

@antblood antblood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: 0 of 2 files reviewed, 19 unresolved discussions (waiting on @harshil-goel, @manishrjain, @martinmr, @pawanrawal, and @vvbalaji-dgraph)


systest/license/license_test.go, line 71 at r4 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

not needed if its false always

Done.


systest/license/license_test.go, line 74 at r4 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

not needed if its nil always

Done.


systest/license/license_test.go, line 110 at r4 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

We should also compare the error message returned to the user. Also, check errors that are returned from json.Unmarshal

Done.


systest/license/license_test.go, line 124 at r4 (raw file):

Previously, pawanrawal (Pawan Rawal) wrote…

check error here

Done.

Copy link
Contributor

@ashish-goswami ashish-goswami left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: 0 of 2 files reviewed, 21 unresolved discussions (waiting on @antblood, @harshil-goel, @manishrjain, @martinmr, @pawanrawal, and @vvbalaji-dgraph)


systest/license/license_test.go, line 1 at r5 (raw file):

package main

Dgraph license is missing here.


systest/license/license_test.go, line 77 at r5 (raw file):

func TestEnterpriseLicense(t *testing.T) {
	t.Log("Running TestEnterpriseLicense\n")

I think this can be removed as something like this already printed while running tests.

Copy link
Contributor

@ashish-goswami ashish-goswami left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 1 of 3 files at r4, 1 of 1 files at r5.
Reviewable status: all files reviewed, 21 unresolved discussions (waiting on @antblood, @harshil-goel, @manishrjain, @martinmr, @pawanrawal, and @vvbalaji-dgraph)

Copy link
Contributor

@martinmr martinmr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewable status: all files reviewed, 21 unresolved discussions (waiting on @antblood, @harshil-goel, @manishrjain, @martinmr, @pawanrawal, and @vvbalaji-dgraph)

Copy link
Contributor Author

@antblood antblood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewable status: all files reviewed, 21 unresolved discussions (waiting on @ashish-goswami, @harshil-goel, @manishrjain, @martinmr, @pawanrawal, and @vvbalaji-dgraph)


systest/license/license_test.go, line 1 at r5 (raw file):

Previously, ashish-goswami (Ashish Goswami) wrote…

Dgraph license is missing here.

Done.


systest/license/license_test.go, line 77 at r5 (raw file):

Previously, ashish-goswami (Ashish Goswami) wrote…

I think this can be removed as something like this already printed while running tests.

Done.

@antblood antblood merged commit 61363d1 into master Jul 16, 2020
dna2github pushed a commit to dna2fork/dgraph that referenced this pull request Jul 18, 2020
Created a file for testing if the license is applied correctly. It contains 3 tests.

1. Applying expired license, this will get applied but it should not be enabled and user information should be extracted
2. Applying invalid license, this won't get applied
3. Applying empty license, this also won't get applied


Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io>
antblood added a commit that referenced this pull request Aug 18, 2020
Created a file for testing if the license is applied correctly. It contains 3 tests.

1. Applying expired license, this will get applied but it should not be enabled and user information should be extracted
2. Applying invalid license, this won't get applied
3. Applying empty license, this also won't get applied


Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io>
(cherry picked from commit 61363d1)
antblood added a commit that referenced this pull request Aug 18, 2020
Created a file for testing if the license is applied correctly. It contains 3 tests.

1. Applying expired license, this will get applied but it should not be enabled and user information should be extracted
2. Applying invalid license, this won't get applied
3. Applying empty license, this also won't get applied


Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io>
(cherry picked from commit 61363d1)
antblood added a commit that referenced this pull request Aug 18, 2020
Created a file for testing if the license is applied correctly. It contains 3 tests.

1. Applying expired license, this will get applied but it should not be enabled and user information should be extracted
2. Applying invalid license, this won't get applied
3. Applying empty license, this also won't get applied


Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io>
(cherry picked from commit 61363d1)
antblood added a commit that referenced this pull request Oct 2, 2020
Created a file for testing if the license is applied correctly. It contains 3 tests.

1. Applying expired license, this will get applied but it should not be enabled and user information should be extracted
2. Applying invalid license, this won't get applied
3. Applying empty license, this also won't get applied

Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io>
(cherry picked from commit 61363d1)
antblood added a commit that referenced this pull request Oct 2, 2020
Created a file for testing if the license is applied correctly. It contains 3 tests.

1. Applying expired license, this will get applied but it should not be enabled and user information should be extracted
2. Applying invalid license, this won't get applied
3. Applying empty license, this also won't get applied

Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io>
(cherry picked from commit 61363d1)
antblood added a commit that referenced this pull request Oct 2, 2020
Created a file for testing if the license is applied correctly. It contains 3 tests.

1. Applying expired license, this will get applied but it should not be enabled and user information should be extracted
2. Applying invalid license, this won't get applied
3. Applying empty license, this also won't get applied

Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io>
(cherry picked from commit 61363d1)
antblood added a commit that referenced this pull request Oct 3, 2020
Enterprise License unit tests
antblood added a commit that referenced this pull request Oct 3, 2020
Enterprise License unit tests
antblood added a commit that referenced this pull request Oct 3, 2020
Enterprise License unit tests
abhimanyusinghgaur added a commit that referenced this pull request Oct 15, 2020
* fix(docs): fix URL paths and make deploy page to not show blank page (#6239)

* Fix URL paths (#6189)

(cherry picked from commit 4ee4af8)

* fix deploy page to not show blank page

(cherry picked from commit 89e012b)

Co-authored-by: Apoorv Vardhan <vardhanapoorv@users.noreply.github.com>

* feat(docs): Port GraphQL docs to v20.07 (#6202)

* Port GraphQL docs to v20.07

This reverts commit 8db61a3.

Co-authored-by: Pawan Rawal <pawan0201@gmail.com>

* doc: Setting the canonical path to each docs without version (#6249)

Passing a few variables to the hugo theme

* fix: remove unnecessary curly braces from doc (#6256)

Co-authored-by: Sankalan Parajuli <sankalan.13@gmail.com>

* fix(Dgraph): Stop forcing RAM mode for the write-ahead log. (#6142) (#6259)

This change also adds a way to set the table and value log loading modes for the w directory
independently of the values for the p directory.

Fixes DGRAPH-1898.

(cherry picked from commit 5f5aa9c)

* Enabling Discuss comments

* Fix(Dgraph): Fix how visited nodes are detected in recurse queries. (#6272) (#6276)

Currently a node is marked as visited if it's been visited before AND
has been visited from the same source UID. In dense graphs, the second
condition leads to exponential growth of the data and to other issues
such as responses that are too big to encode.  Removing this condition
fixes the issue.

Fixed tests and verified the new output makes sense.

Fixes DGRAPH-2337

(cherry picked from commit 3638c12)

* fix panicwrap parent check (#6264) (#6299)

(cherry picked from commit 52e136f)

* release/v20.07 - add --cache_mb and --cache_percentage flag (#6286)

* fix(Dgraph): Sort manifests by BackupNum in file handler. (#6263) (#6279)

This was being done in the S3 handler but not in the file handler.
By default the paths are sorted by the filename but sorting by backup
number is more robust.

(cherry picked from commit df63d6c)

* fix(ACL Query): Fixes queries which use variable at the top level (#6290) (#6298)

(cherry picked from commit 6f883a4)

* Revert "Fix(Dgraph): Fix how visited nodes are detected in recurse queries. (#6272) (#6276)" (#6306)

This reverts commit 9df9f92.

* fix(Dgraph): Don't store start_ts in postings. (#6213)

Clear out the start_ts field in the postings of deltas before they are
marshalled to avoid storing that field in disk. This field is only meant
to be used during in-memory processing.

Related to https://discuss.dgraph.io/t/start-ts-not-being-cleared-before-postings-are-written-to-disk/9146

(cherry picked from commit fbbd731)

* docs: add info about __typename in custom DQL (#6248) (#6273)

Reference: [Discuss Post](https://discuss.dgraph.io/t/custom-dql-typename-bug-strange-behaviour/9341)
(cherry picked from commit 0faae40)

* fix(GraphQL): fixes unexpected fragment behaviour (#6228) (#6274)

Fixes: #5516
Fixes: [Discuss Issue](https://discuss.dgraph.io/t/graphql-fragments-generates-unexpected-behaviour/7470)
(cherry picked from commit dcce8bf)

* fix(task): Return error on closed DB (#6075) (#6320)

Fixes DGRAPH-2181

The queries in dgraph are processed in separate goroutines. The pstore badger DB
could be closed while the query was being processed. This causes panics such as
```
panic: runtime error: invalid memory address or nil pointer dereference
	panic: Unclosed iterator at time of Txn.Discard.
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x11d2311]

goroutine 19298 [running]:
github.com/dgraph-io/badger/v2.(*Txn).Discard(0xc05586bc20)
	/go/pkg/mod/github.com/dgraph-io/badger/v2@v2.0.1-rc1.0.20200718033852-37ee16d8ad1c/txn.go:517 +0xc1
panic(0x19e26a0, 0x2988560)
	/usr/local/go/src/runtime/panic.go:969 +0x166
github.com/dgraph-io/badger/v2/skl.(*Skiplist).IncrRef(...)
```
This PR attempts to reduce the number of such crashes. This PR doesn't fix the actual
issue but it tries to reduce the probability of such crashes by checking if badger is not
closed before accessing it.

An ideal fix would be to stop all the goroutines started by dgraph while closing DB so
that we don't try to read from a closed badger DB.

(cherry picked from commit 3cea0fe)

* fix(Dgraph): make backups cancel other tasks (#6152) (#6243)

Backups should be added to the list of tasks (rollups, snapshots, etc)
that are managed by Dgraph so that rollups and other tasks are paused
during the backup.

(cherry picked from commit ea15b66)

* Update badger to v2.2007.2 (#6294)

This badger update brings the following changes from badger.

https://github.com/dgraph-io/badger/commits/b41e77a Fix(cleanup): Avoid truncating in value.Open on error
https://github.com/dgraph-io/badger/commits/a0d4903 fix(cleanup): Do not close cache before compaction

* posting list fixes. (#6303) (#6332)

(cherry picked from commit deded8b)

* fix: Online Restore honors credentials passed in (#6295) (#6302)

(cherry picked from commit a8a6e85)

Co-authored-by: Tejas Dinkar <tejas@gja.in>

* (release/v20.07) Fix(Dgraph): Add a lock to backups to process one request at a time. (#6339)

It's possible that two requests reach the server around the same time
and send a requests to the alphas with the same backupNum. This could
lead to issues further down the line.

Related to DGRAPH-2295

(cherry picked from commit 5b79260)

* (release/v20.07) fix(Dgraph): Add flags to set table and vlog loading mode for zero. (#6342)

Related to DGRAPH-2189

(cherry picked from commit bf79999)

* proto fix needed for PR #6331 (#6337) (#6346)

(cherry picked from commit 844f34d)

* chore(GraphQL): remove skipped tests (#6078) (#6347)

Fixes DGRAPH-2357.
Fixes GRAPHQL-562.

(cherry picked from commit 91e28d5)

* test: add --build option for docker-compose up in test script (#6348) (#6352)

Fixes DGRAPH-2356
This PR fixes random GraphQL custom logic e2e test failures in TeamCity by requiring docker-compose to always build images before starting the containers, as the mock server in custom logic tests can change any time.

(cherry picked from commit 536b317)

* cherry-pick v20.07: fix(GraphQL): incorrect generatedSchema in updateGQLSchema (#6349) (#6354)

* fix(GraphQL): incorrect generatedSchema in updateGQLSchema (#6349)

This PR fixes the behaviour where Dgraph schema was being given as generatedSchema field in updateGQLSchema request. Now, the newly generated complete GraphQL schema is given as generatedSchema.

(cherry picked from commit 0b8681c)

# Conflicts:
#	graphql/e2e/schema/schema_test.go

* (release/v20.07) fix(Dgraph): add flag to set up compression in zero. (#6355)

(cherry picked from commit 254bd29)

* fix(dgraph): sentry nil pointer check (#6372) (#6374)

(cherry picked from commit 7857fa2)

* doc: Adding a note on backend modes

* doc: Adding a security page on slash

* chore(Docs): Update graph-links.md to remove a duplicate word. (#6378)

remove duplicate word "doesn't" from description of two-way edges.

* doc: One-Click Documentation for Slash (#6381 #6383) (#6384)

(cherry picked from commit e165710)

* Use z.Closer instead of y.Closer (#6394) (#6399)

The closer type was moved from badger/y to ristretto/z in dgraph-io/ristretto#191

(cherry picked from commit 0287838)

* (release/v20.07) docs: Add section on automating backups. (#6405)

(cherry picked from commit 0fab710)

* relase/v20.07 - Fix(Alpha): MASA: Make Alpha Shutdown Again (#6313)  (#6402)

* Fix(Alpha): MASA: Make Alpha Shutdown Again (#6313)

This PR removes the usage of context.Background() in groups.go and ensures that all the various goroutines exit as intended.

Changes:
* Shutdown SubscribeForUpdates correctly.
* Fix up all the closing conditions.
* Consolidate updaters into one closer
* Update Badger to master
* fix(build): Update ResetAcl args for OSS build.
* chore: Remove TODO comment.

Co-authored-by: Daniel Mai <daniel@dgraph.io>
(cherry picked from commit f1941b3)

* fix(shutdown): Force exit if CTRL-C is caught before initialization (#6359) (#6407)

Forcefully kill alpha if CTRL-C is caught before node initialization completes.

Fixes DGRAPH-2377

Co-authored-by: Manish R Jain <manish@dgraph.io>
(cherry picked from commit 41c2052)

* Update advanced-queries.md

* fix(Query) Fix Star_All delete query when used with ACL enabled (#6331) (#6336)

* docs: Add netlify deploy docs to slash docs (#6425) (#6428)

(cherry picked from commit d94e5da)

* docs: add whitelisting issue error in troubleshoot page to help the community. (#6422)

Fixes DGRAPH-2317. Issue is with v20.07 we restrict the admin operations from alpha instances only. But this hasn't been mentioned during the deployment process in any examples. Hence community is running into issues like Unauthorized IP address X.X.X.X. This doc addition in troubleshoot page will help the community members if they run into this issue.

* Add badger to docs (#6430)

* fix(bulk): Batch list in bulk loader to avoid panic (#6446)

* Changelog and Codename (#6488)

* fix: Regenerate protos.

The different generated proto file makes the release.sh script fail
with "Generated protos different in release."

Changes
* Called "make regenerate"

* fix(GraphQL): fix introspection completion bug (#6385) (#6389)

Fixes GRAPHQL-673.
This PR fixes the issue where introspection queries would break if there were two types implementing same interface and having fields with same name in those two types and that repeating field is also a field in introspection query.
This was introduced after #6228.

(cherry picked from commit 243a336)

* fix(GraphQL): don't generate orderable enum value for list fields (#6392) (#6413)

Fixes GRAPHQL-650.

For this user schema:
```
type Starship {
        id: ID!
        name: String! @search(by: [term])
        length: Int64
        tags: [String]
        createdAt: DateTime
}
```
we were generating complete GraphQL schema with StarshipOrderable as:
```
enum StarshipOrderable {
  name
  length
  tags
  createdAt
}
```
that would cause a DQL query to be formed which errors out, see this GraphQL query:
```
query {
  queryStarship(order: {asc: tags}) {
    id
    name
    length
    tags
  }
}
```
It gives back:
```
{
  "errors": [
    {
      "message": "Dgraph query failed because Dgraph execution failed because : Sorting not supported on attr: Starship.tags of type: [scalar]"
    }
  ],
  "data": {
    "queryStarship": []
  }
}
```
which means we should not allow even list of scalar along with object types in orderable. Only scalar field should be allowed in orderable. So, the correct orderable should be without tags field:
```
enum StarshipOrderable {
  name
  length
  createdAt
}
```

This PR fixes the above bug.

(cherry picked from commit dc66617)

* feat(Dgraph): Add separate compression flag for z and wal dirs (#6401) (#6421)

* Add separate compression flag for z and wal dirs

* Address comments

* Address comments

(cherry picked from commit 601cc3b)

* docs: Update list documentation (#6455) (#6461)

Fixes GRAPHQL-657
Fixes [Discuss Issue](https://discuss.dgraph.io/t/documentation-that-arrays-work-as-a-set-not-storing-duplicates/9590)

(cherry picked from commit 07cff6f)

* cherry-pick v20.07: fix(GraphQL): Generate correct schema when no orderable field in a ty… (#6460)

fix(GraphQL): Generate correct schema when no orderable field in a type (#6456)

Fixes GRAPHQL-650
It was introduced after #6392.

(cherry picked from commit 8e4d121)

* (release/v20.07) test: retry getting client in TestNodes. (#6482)

(cherry picked from commit 512c69b)

* (release/v20.07) fix(Dgraph): fix bug when deleting and adding to a single UID predicate in the same transaction. (#6449)

The iterate method does not account for delete postings in the same transaction
so they have to be preserved manually. Otherwise, if a deletion and an add operation
to the same uid happen in the same predicate, the deletion operation is not
applied.

Fixes DGRAPH-1309

(cherry picked from commit 56c8f3c)

* Fix(docs): Fix subsection references in docs (#6439) (#6516)

* Fix references in docs

* Fix mutations/language-rdf-types.md reference

(cherry picked from commit 027db53)

* Fix(increment): Fix readTs less than minTs (#6317) (#6517)

Use the readTs for reads via LocalCache. This would ensure that users don't get a lot of ReadTs less than MinTs errors. This PR also fixes the flaky test in increment_test.go.

This has a side-effect on posting list cache. That cache would need to understand versioning to be effective.

Co-authored-by: Daniel Mai <daniel@dgraph.io>
(cherry picked from commit 2de4675)

Co-authored-by: Manish R Jain <manish@dgraph.io>

* fix(GraphQL): Fix order and offset in auth queries. (#6221) (#6366)

* Fix order and offset in auth queries.

* Remove order from top-level auth filter.

(cherry picked from commit a5bfd4a)

* fix(GraphQL): Fix query rewriting for auth delete when deleting types with inverse field. (#6350) (#6524)

* Fix query rewriting for auth delete when deleting types with inverse field.

(cherry picked from commit be9ebd0)

* cherry-pick v20.07: fix(GraphQL): Link xids properly if there are duplicate xids within t… (#6521)

Fixes GRAPHQL-641

See https://discuss.dgraph.io/t/residual-issue-linking-grandchild-in-mutation-with-custom-ids/9613 for the issue details.

If the same nested xid was used as a leaf node within an addType mutation, it was only linked the first time that it was used and not subsequently for later objects. This change fixes that by adding the linkage mutations and only removing the mutation which was creating the object with the xid again.

(cherry picked from commit 12b2716)

# Conflicts:
#	graphql/e2e/directives/dgraph_directives_test.go
#	graphql/e2e/normal/normal_test.go
#	graphql/resolve/add_mutation_test.yaml

* fix(GraphQL): fix internal error when doing GraphQL schema introspection after drop all (#6268) (#6525)

This PR fixes the "Internal error" response when querying the GraphQL schema after performing drop_all operation.

(cherry picked from commit d3bee33)

* fix(GraphQL): fix for deletion on interfaces with no non Id field (#6387) (#6417)

Fixes GRAPHQL-655.

For this user schema (interface have no non-Id field):

```
interface A {
   name: String! @id
}

type B implements A {
  age: Int!
}
```
the following delete mutation
```
mutation{
  deleteA(filter:{name:{eq: "xyz"}}){
    a{
      name
    }
  }
}
```
was resulting in the following error:
```
{
  "errors": [
    {
      "message": "Internal Server Error - a panic was trapped.  This indicates a bug in the GraphQL server.  A stack trace was logged.  Please let us know by filing an issue with the stack trace."
    }
  ]
}
```
This PR fixes this bug and now deleting can be performed successfully.

(cherry picked from commit 742259b)

* fix(GraphQL): fix  restoreStatus query with query variables (#6414) (#6424)

Fixes GRAPHQL-642.

For this restoreStatus query using variable
```
query restoreStatus($restoreId: Int!) {
	restoreStatus(restoreId: $restoreId) {
		status
		errors
	}
}
```
was giving this panic
```
panic: interface conversion: interface {} is json.Number, not int64.
```
Whereas the expected result should be
```
{
  "data": {
    "restoreStatus": {
      "status": "UNKNOWN",
      "errors": []
    }
  },
  "extensions": {}
}
```
This PR fixes this panic, Now `resolveStatus` with or without variable works fine.

(cherry picked from commit 45afae9)

* fix(GraphQl): fix errors from authorization examples given in docs. (#6330) (#6522)

This PR fix errors from authorization examples given in docs.

(cherry picked from commit 0f3bfb1)

* Changes github.com/dgraph-io/graphql-transport-ws version (#6462) (#6529)

(cherry picked from commit 5aca255)

* Remove auth error from mutation. (#6532)

* fix(GraphQL): This PR add schema cleaning in GraphQL. (#6523)

* Fix(GraphQL): This PR add schema cleaning in GraphQL. (#6427)

Fixes GRAPHQL-572

This PR adds schema cleaning in GraphQL which will remove empty links and types from generated GraphQL schema recursively.

(cherry picked from commit bec5322)

* fix(GraphQL): Fix squashIntoObject so that results are correctly merged (#6416) (#6530)

While creating the add mutation, we weren't squashing different bits correctly which led to the incorrect mutation being sent to Dgraph. This change modifies the set to append for []interface{} to fix that.

Fixes GRAPHQL-679

(cherry picked from commit 816a08f)

# Conflicts:
#	graphql/e2e/common/mutation.go
#	graphql/e2e/directives/schema.graphql
#	graphql/e2e/normal/schema.graphql
#	graphql/resolve/schema.graphql

* fix(GraphQL): Fix cascade with auth query when RBAC is false (#6444) (#6535)

* fix(GraphQL): Fix cascade with auth query when RBAC is false (#6444)

* Fix cascade with auth query.

* Added additional E2E test.

(cherry picked from commit 8c518d4)

* fix(GraphQL): Hide info when performing mutation on id field with auth rule. (#6391) (#6534)

* fix(GraphQL): Hide info when performing mutation on id field with auth rule. (#6391)

* Hide info when performing mutation on id field with auth rule.

(cherry picked from commit 5c33428)

* fix(GraphQL): This PR adds documentation for Scalar DateTime type. (#6373) (#6554)

Fixes GRAPHQL-631
This PR adds documentation for Scalar DateTime type.

(cherry picked from commit 91c81ca)

* Revert "fix(GraphQL): This PR add schema cleaning in GraphQL. (#6523)" (#6559)

This reverts commit 812a6f0.

# Conflicts:
#	graphql/schema/testdata/schemagen/output/filter-cleanSchema-all-empty.graphql
#	graphql/schema/testdata/schemagen/output/filter-cleanSchema-circular.graphql
#	graphql/schema/testdata/schemagen/output/filter-cleanSchema-custom-mutation.graphql
#	graphql/schema/testdata/schemagen/output/filter-cleanSchema-directLink.graphql

* Update index.md (#6566) (#6567)

(cherry picked from commit 512c0a9)

* perf(raft): Use raft storage in managedmode (#6457) (#6547)

This PR changes how we use badger in the wal store. Currently, we run
badger in normal mode for `w` and `zw` store. We've seen up to 900K
entries for the same hard state (hs) key in `zw` store. These duplicate
keys cause spikes in read latencies in `zw`. The `w` store has more
compactions compared to `zw` store and so it has lesser stale data and
thus lesser spikes in read latencies.

The fix here is to open the `w` and `zw` directories in managed mode and
perform all writes on the same timestamp (max version in the db). This
leads to close to 0 duplicates in the store.

This PR also fixes the raft leader election issue which is a result of high
read latencies.

(cherry picked from commit 6882e37)

* chore(query): Return content length header for queries (#6469) (#6480)

This PR adds `Content-Length` header to the response of queries sent via HTTP.

Fixes DGRAPH-1675

(cherry picked from commit ed63786)

* fix(bulkLoader): Use flags for cache (#6322) (#6467)

Bulk loader uses caches in compression and this PR adds flags to make it
configurable.

Bulk loader was setting compressionLevel but not the compression option.
As a result of this, badger wasn't compressing any data. This PR fixes
this setting compression if compressionLevel is greater than 0.

(cherry picked from commit 99341dc)

* docs: add documentation for clone backend (#6622)

* (release/v20.07): feat(Dgraph): add utility to export backup data. (#6590)

* (release/v20.07): feat(Dgraph): add utility to export backup data. (#6550)

This utility allows to take a single backup (full or incremental) and export the data
and the schema inside it to RDF. Encrypted backups are supported.

Fixes DGRAPH-2465

(cherry picked from commit 369a5c1)

* Remove destination field.

* (release/v20.07) build: Makefile to build Dgraph inside docker container. (#6601)

The script is useful for situations in which the host system creates a
binary that's incompatible with the docker container (e.g different
versions of glibc).

(cherry picked from commit c789738)

* (release/v20.07) Pause rollups during snapshot streaming. (#6611)

(cherry picked from commit b777913)

* (release/v20.07) test: Fix flakiness in group_delete_test.go (#6626)

The test and utility were assuming that node 1 was in group 1, node 2 in
group 2, and so on. This is not true all the time, hence the flakiness.

Fixing the code to not make this assumptions as well as retrying the
DropAll in NodesSetup to ensure all the nodes are up before proceeding
with the test.

(cherry picked from commit 36638a7)

* CID never created if Zero stops early after first init (#5719) (#6637)


(cherry picked from commit 09ef988)

* test: Enterprise License tests (#5822) (#6640)

Enterprise License unit tests

* Pagination param "after" does not work when using func: uid(v) (#6365) (#6634)

fixed : Pagination param "after" does not work when using func: uid(v)

* (release/v20.07) Update nextRaftId when starting a node with a raftId > 0. (#6597)


(cherry picked from commit a4f21f4)

* release/v20.07 - fix(gqlgen): Update gqlgen in go.mod (#6594) (#6646)

* fix(gqlgen): Update gqlgen in go.mod (#6594)

* Add tests for querying alias and attribute on interface

* Requested changes

(cherry picked from commit f19d882)

* Fix test

* Fix(GraphQL): fix object Linking with `hasInverse` (#6557) (#6648)

This PR fixes incorrect linking of objects with `hasInverse`. For example
For this schema, `type Country` has field `state` with `hasInverse` predicate.
```
type Country {
        id: ID!
        name: String! @search(by: [trigram, hash])
        states: [State] @hasInverse(field: country)
}

type State {
        id: ID!
        xcode: String! @id @search(by: [regexp])
        name: String!
	capital: String
	country: Country
}
```
And for the following mutation
```
mutation addCountry($input: [AddCountryInput!]!) {
			addCountry(input: $input) {
			  country {
				name
				states{
				  xcode
				  name
				  country{
					name
				  }
				}
			  }
			}
		  }
```
with `input`:
```
{
	"input": {
		"name": "A Country",
		"states": [
			{
				"xcode": "abc",
				"name": "Alphabet"
			},
			{
				"xcode": "def",
				"name": "Vowel",
				"country": {
					"name": "B country"
				}
			}
		]
	}
}
```
should result in
```
{
		"addCountry": {
		  "country": [
			{
			  "name": "A country",
			  "states": [
				{
				  "country": {
					"name": "A country"
				  },
				  "name": "Alphabet",
				  "xcode": "abc"
				},
				{
				  "country": {
					"name": "A country"
				  },
				  "name": "Vowel",
				  "xcode": "def"
				}
			  ]
			}
		  ]
		}
	  }`
```
whereas the incorrect output was
```
{
		"addCountry": {
		  "country": [
			{
			  "name": "A country",
			  "states": [
				{
				  "country": {
					"name": "A country"
				  },
				  "name": "Alphabet",
				  "xcode": "abc"
				},
				{
				  "country": {
					"name": "B country"
				  },
				  "name": "Vowel",
				  "xcode": "def"
				}
			  ]
			}
		  ]
		}
	  }`
```
This PR fixes this issue.

(cherry picked from commit e496467)

* doc: update /admin endpoint documentation. (#6415) (#6527)

This PR updates documentation for the `/admin` endpoint operation and  `restore` operation.

(cherry picked from commit 288eafb)

* fix(GraphQL Query): Remove extra fields when querying interfaces (#6596) (#6647)

(cherry picked from commit cec5567)
This PR fixes: GRAPHQL-638

* fix (graphql): disallowing field names with as (#6645)

This is related to GRAPHQL-564. as is Dgraph reserved keyword. It's being used by DQL to identify variables. This PR is to restrict the naming for fields with the name as.

* (release/v20.07) docs: Update restore documentation. (#6653)

(cherry picked from commit 014ee10)

* fix(Query): Fix wrong path response for k-shortest paths (#6437) (#6654)

The k-shortest path query sometimes returns a wrong response. The cause
of this issue is inappropriate use of sync.pools. Whenever a valid path
is found a new variable is assigned with this route and that variable is
appended in the kroutes (which is then parsed to produce response).
But the route is also put in the sync.pool. Since the route is a
structure containing a pointer to slice along with other fields, when
it is fetched back from the pool in future then any mutation on this fetched
route causes modification in the kroutes, resulting in garbage response.

This issue is fixed by making a copy of the route struct rather
than just assigning it to a variable, which causes a shallow copy.

(cherry picked from commit 4792d8b)

* Updater badger to 5e3d4b9 (#6669)

* Break out if g.Ctx is done (#6675)

(cherry picked from commit a8a47af)

Co-authored-by: Manish R Jain <manish@dgraph.io>

* fix(Auth): fix Poor-man's auth for admin operations (#6660) (#6686)

Fixes DGRAPH-2419
Fixes [Discuss Issue](https://discuss.dgraph.io/t/acl-login-will-fail-if-auth-token-enabled-in-v20-07-0/10044)

This PR fixes Poor-man's auth for following endpoints:
* `/login`
* `/admin`

(cherry picked from commit 4fc328d)

# Conflicts:
#	graphql/e2e/common/common.go

* Merge branch 'release/v20.07' of github.com:dgraph-io/dgraph into abhimanyu/release-slash-merge

# Conflicts:
#	dgraph/cmd/alpha/run.go
#	dgraph/cmd/bulk/reduce.go
#	dgraph/cmd/zero/raft.go
#	edgraph/server.go
#	go.mod
#	go.sum
#	graphql/e2e/common/common.go
#	graphql/e2e/common/fragment.go
#	graphql/e2e/common/mutation.go
#	graphql/e2e/common/query.go
#	graphql/e2e/custom_logic/custom_logic_test.go
#	graphql/e2e/directives/schema_response.json
#	graphql/e2e/normal/schema_response.json
#	graphql/e2e/schema/generatedSchema.graphql
#	graphql/resolve/resolver.go
#	graphql/schema/testdata/schemagen/output/type-without-orderables.graphql
#	protos/pb/pb.pb.go
#	wiki/content/enterprise-features/binary-backups.md
#	wiki/content/graphql/admin/index.md
#	wiki/content/graphql/schema/reserved.md

* fix tests

Co-authored-by: Naman Jain <naman@dgraph.io>
Co-authored-by: Apoorv Vardhan <vardhanapoorv@users.noreply.github.com>
Co-authored-by: Pawan Rawal <pawan0201@gmail.com>
Co-authored-by: Tejas Dinkar <tejas@gja.in>
Co-authored-by: Sankalan Parajuli <sankalan@dgraph.io>
Co-authored-by: Sankalan Parajuli <sankalan.13@gmail.com>
Co-authored-by: Martin Martinez Rivera <martinmr@dgraph.io>
Co-authored-by: parasssh <paras@dgraph.io>
Co-authored-by: Anurag <anurag@dgraph.io>
Co-authored-by: Ibrahim Jarif <ibrahim@dgraph.io>
Co-authored-by: Martin Martinez Rivera <mrtnz.rvr@gmail.com>
Co-authored-by: Tyler <haikusw@gmail.com>
Co-authored-by: aman bansal <amanbansal@dgraph.io>
Co-authored-by: Harshil Goel <54325286+harshil-goel@users.noreply.github.com>
Co-authored-by: Daniel Mai <daniel@dgraph.io>
Co-authored-by: vmrajas <rajas@dgraph.io>
Co-authored-by: Manish R Jain <manish@dgraph.io>
Co-authored-by: Arijit Das <arijit@dgraph.io>
Co-authored-by: minhaj-shakeel <minhaj@dgraph.io>
Co-authored-by: Jatin Dev <64803093+JatinDevDG@users.noreply.github.com>
Co-authored-by: Akash Jain <akashjain971@gmail.com>
Co-authored-by: Neeraj Battan <66044626+antblood@users.noreply.github.com>
Co-authored-by: aman bansal <bansalaman2905@gmail.com>
Co-authored-by: Ahsan Barkati <ahsanbarkati@gmail.com>
@joshua-goldstein joshua-goldstein deleted the neeraj/LicenseTesting branch August 11, 2022 20:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants