BREAKING: fix json marshal unmarshal for namespace > 127 (#7810,#7838,#8091) #8599 #8602
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
harshil-goel
requested review from
akon-dey,
MichelDiz,
matthewmcneely,
skrdgraph,
darkn3rd,
meghalims and
billprovince
as code owners
|
|
mangalaman93
added
the
slash-to-main
harshil-goel
force-pushed
the
harshil/cp
branch
from
53b2773 to
3fccd72
Compare
harshil-goel
changed the base branch from
harshil/manifest_bug_fix
to
anurag/cherry-pick-drop-data
harshil-goel
changed the title
harshil-goel
changed the title
mangalaman93
previously approved these changes
Jan 16, 2023
@harshil-goel Was this accomplished in this PR? Have all use-cases been addressed with this namespace formatting change (migration, etc)? |
mangalaman93
force-pushed
the
anurag/cherry-pick-drop-data
branch
4 times, most recently
from
4e1118c to
6c16b0b
Compare
We used to store predicate as <namespace>|<attribute> (pipe | signifies concatenation). We store this as a string. <namespace> is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph: Live loader Backup and List Backup Http clients and Ratel Fix: Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte. New Format: <anmespace>-<attribute> (- is the hyphen literal)
With #7810 change, we changed the format of the predicate. We missed updating the schema and predicate. This PR fixes it.
There is an issue in ExtractNamespaceFromPredicate. The issue is the parsing was done assuming ns in <ns>-<attr> to be decimal (actually it is hexadecimal). This leads to the following issues. A predicate a-name, it was skipped. A predicate 11-name was parsed as namespace 11, actually it is namespace 17 (0x11).
harshil-goel
force-pushed
the
harshil/cp
branch
from
3fccd72 to
fed155a
Compare
mangalaman93
approved these changes
Jan 30, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We used to store predicate as | (pipe | signifies concatenation). We store this as a string. is 8 bytes uint64, which when marshaled to JSON bytes mess up the predicate. This is because for the namespace greater than 127, the UTF-8 encoding might take up several bytes (also if the mapping does not exist, then it replaces it with some other rune). This affects three identified places in Dgraph:
Live loader
Backup and List Backup
Http clients and Ratel
Fix:
Fix is to have a UTF-8 string when dealing with JSON. A better idea is to use UTF-8 string even for internal operations. Only when we read/write to badger we convert it into the format of the byte.
New Format: - (- is the hyphen literal)