Skip to content

handle manifest version logic, update manifest version to 2105 #7825

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 17, 2021

Conversation

NamanJain8
Copy link
Contributor

@NamanJain8 NamanJain8 commented May 17, 2021

The backward compatibility of the backup's manifest was broken by #7810, although the tool was added (#7815) that enables smooth migration of manifest.
This PR makes backup backward compatible, by updating the manifest(in-memory) after reading.


This change is Reviewable

@github-actions github-actions bot added the area/enterprise Related to proprietary features label May 17, 2021
Copy link
Contributor

@manishrjain manishrjain left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:lgtm:

Reviewed 34 of 34 files at r1.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on @NamanJain8 and @vvbalaji-dgraph)


ee/updatemanifest/run.go, line 126 at r1 (raw file):

	// Update the master manifest with the changes for drop operations and group predicates.
	for _, manifest := range masterManifest.Manifests {
		if manifest.Version == 2103 {

<= ??


worker/backup_ee.go, line 551 at r1 (raw file):

	}

	manifest, err := GetManifestWithoutUpgrade(handler, uri)

GetManifestNoUpgrade


worker/backup_manifest.go, line 169 at r1 (raw file):

func parseNsAttr(attr string) (uint64, string, error) {
	if strings.ContainsRune(attr, replacementRune) {
		return 0, "", errors.New("replacement char found")

Improve the error.


worker/backup_manifest.go, line 225 at r1 (raw file):

		}
	case 2105:
		// pass

default case?


worker/backup_manifest.go, line 266 at r1 (raw file):

// GetManifestWithoutUpgrade returns the master manifest using the given handler and uri.
func GetManifestWithoutUpgrade(h UriHandler, uri *url.URL) (*MasterManifest, error) {

GetManifestNoUpgrade

@NamanJain8 NamanJain8 merged commit 83a0c53 into release/v21.03-slash May 17, 2021
@NamanJain8 NamanJain8 deleted the naman/manifest-version branch May 17, 2021 18:37
Copy link
Contributor Author

@NamanJain8 NamanJain8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed 26 of 34 files at r1.
Dismissed @manishrjain from 5 discussions.
Reviewable status: 30 of 34 files reviewed, all discussions resolved


ee/updatemanifest/run.go, line 126 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

<= ??

This is for a specific tool, so we need to check only for version 2103.


worker/backup_ee.go, line 551 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

GetManifestNoUpgrade

Done.


worker/backup_manifest.go, line 169 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

Improve the error.

Done.


worker/backup_manifest.go, line 225 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

default case?

2105 is the default case.


worker/backup_manifest.go, line 266 at r1 (raw file):

Previously, manishrjain (Manish R Jain) wrote…

GetManifestNoUpgrade

Done.

NamanJain8 added a commit that referenced this pull request May 24, 2021
…o 2105 (#7825)

The backward compatibility of the backup's manifest was broken by #7810, although the tool was added (#7815) that enables smooth migration of manifest.
This PR makes backup backward compatible, by updating the manifest(in-memory) after reading.

(cherry picked from commit 83a0c53)
harshil-goel pushed a commit that referenced this pull request Jan 16, 2023
…o 2105 (#7825)

The backward compatibility of the backup's manifest was broken by #7810, although the tool was added (#7815) that enables smooth migration of manifest.
This PR makes backup backward compatible, by updating the manifest(in-memory) after reading.
harshil-goel pushed a commit that referenced this pull request Jan 30, 2023
…o 2105 (#7825)

The backward compatibility of the backup's manifest was broken by #7810, although the tool was added (#7815) that enables smooth migration of manifest.
This PR makes backup backward compatible, by updating the manifest(in-memory) after reading.
harshil-goel pushed a commit that referenced this pull request Jan 31, 2023
…o 2105 (#7825)

The backward compatibility of the backup's manifest was broken by #7810, although the tool was added (#7815) that enables smooth migration of manifest.
This PR makes backup backward compatible, by updating the manifest(in-memory) after reading.
harshil-goel pushed a commit that referenced this pull request Feb 2, 2023
…o 2105 (#7825)

The backward compatibility of the backup's manifest was broken by #7810, although the tool was added (#7815) that enables smooth migration of manifest.
This PR makes backup backward compatible, by updating the manifest(in-memory) after reading.
harshil-goel pushed a commit that referenced this pull request Feb 3, 2023
We used to store predicate as <namespace>|<attribute> (pipe | signifies concatenation). We store this as a string. <namespace> is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph:

Live loader
Backup and List Backup
Http clients and Ratel
Fix:
Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte.
New Format: <anmespace>-<attribute> (- is the hyphen literal)

fix(restore): update the schema and type from 2103 (#7838)

With #7810 change, we changed the format of the predicate. We missed updating the schema and predicate. This PR fixes it.

fix(state): fix hex to uint64 response of list of namespaces (#8091)

There is an issue in ExtractNamespaceFromPredicate. The issue is the parsing was done assuming ns in <ns>-<attr> to be decimal (actually it is hexadecimal). This leads to the following issues.

A predicate a-name, it was skipped.
A predicate 11-name was parsed as namespace 11, actually it is namespace 17 (0x11).

fix(backup): handle manifest version logic, update manifest version to 2105 (#7825)

The backward compatibility of the backup's manifest was broken by #7810, although the tool was added (#7815) that enables smooth migration of manifest.
This PR makes backup backward compatible, by updating the manifest(in-memory) after reading.

fix(updatemanifest): update the version of manifest after update (#7828)

We were not updating the manifest version after the updation. This PR fixes that.
harshil-goel pushed a commit that referenced this pull request Feb 3, 2023
We used to store predicate as <namespace>|<attribute> (pipe | signifies concatenation). We store this as a string. <namespace> is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph:

Live loader
Backup and List Backup
Http clients and Ratel
Fix:
Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte.
New Format: <anmespace>-<attribute> (- is the hyphen literal)

fix(restore): update the schema and type from 2103 (#7838)

With #7810 change, we changed the format of the predicate. We missed updating the schema and predicate. This PR fixes it.

fix(state): fix hex to uint64 response of list of namespaces (#8091)

There is an issue in ExtractNamespaceFromPredicate. The issue is the parsing was done assuming ns in <ns>-<attr> to be decimal (actually it is hexadecimal). This leads to the following issues.

A predicate a-name, it was skipped.
A predicate 11-name was parsed as namespace 11, actually it is namespace 17 (0x11).

fix(backup): handle manifest version logic, update manifest version to 2105 (#7825)

The backward compatibility of the backup's manifest was broken by #7810, although the tool was added (#7815) that enables smooth migration of manifest.
This PR makes backup backward compatible, by updating the manifest(in-memory) after reading.

fix(updatemanifest): update the version of manifest after update (#7828)

We were not updating the manifest version after the updation. This PR fixes that.
harshil-goel added a commit that referenced this pull request Feb 3, 2023
…t json marshal issues (#8601)

We used to store predicate as <namespace>|<attribute> (pipe | signifies
concatenation). We store this as a string. <namespace> is 8 bytes
uint64, which when marshaled to JSON bytes mess up the predicate. This
is because for the namespace greater than 127, the UTF-8 encoding might
take up several bytes (also if the mapping does not exist, then it
replaces it with some other rune). This affects three identified places
in Dgraph:

- Live loader using guardian of galaxy
- Backup and List Backup
- Http clients and Ratel
- Schema and predicate

Fix:
Fix is to have a UTF-8 string when dealing with JSON. A better idea is
to use UTF-8 string even for internal operations. Only when we
read/write to badger we convert it into the format of the byte.
New Format: <namespace>-<attribute> (- is the hyphen literal)
<namespace> is a string "81" in hex

We also update the manifest version after update. This diff takes care
that older backups are still compatible and can be used to restore.

Contains: 
#7838
#7828
#7825
#7815
#7810
all-seeing-code pushed a commit that referenced this pull request Feb 8, 2023
…t json marshal issues (#8601)

We used to store predicate as <namespace>|<attribute> (pipe | signifies
concatenation). We store this as a string. <namespace> is 8 bytes
uint64, which when marshaled to JSON bytes mess up the predicate. This
is because for the namespace greater than 127, the UTF-8 encoding might
take up several bytes (also if the mapping does not exist, then it
replaces it with some other rune). This affects three identified places
in Dgraph:

- Live loader using guardian of galaxy
- Backup and List Backup
- Http clients and Ratel
- Schema and predicate

Fix:
Fix is to have a UTF-8 string when dealing with JSON. A better idea is
to use UTF-8 string even for internal operations. Only when we
read/write to badger we convert it into the format of the byte.
New Format: <namespace>-<attribute> (- is the hyphen literal)
<namespace> is a string "81" in hex

We also update the manifest version after update. This diff takes care
that older backups are still compatible and can be used to restore.

Contains: 
#7838
#7828
#7825
#7815
#7810
all-seeing-code pushed a commit that referenced this pull request Feb 8, 2023
…t json marshal issues (#8601)

We used to store predicate as <namespace>|<attribute> (pipe | signifies
concatenation). We store this as a string. <namespace> is 8 bytes
uint64, which when marshaled to JSON bytes mess up the predicate. This
is because for the namespace greater than 127, the UTF-8 encoding might
take up several bytes (also if the mapping does not exist, then it
replaces it with some other rune). This affects three identified places
in Dgraph:

- Live loader using guardian of galaxy
- Backup and List Backup
- Http clients and Ratel
- Schema and predicate

Fix:
Fix is to have a UTF-8 string when dealing with JSON. A better idea is
to use UTF-8 string even for internal operations. Only when we
read/write to badger we convert it into the format of the byte.
New Format: <namespace>-<attribute> (- is the hyphen literal)
<namespace> is a string "81" in hex

We also update the manifest version after update. This diff takes care
that older backups are still compatible and can be used to restore.

Contains: 
#7838
#7828
#7825
#7815
#7810
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/enterprise Related to proprietary features
Development

Successfully merging this pull request may close these issues.

2 participants