Trust fails to persist

[joyceerhl]

The hashes stored to trust store in trustNotebook are not the same as the hashes computed when checking isNotebookTrusted. This used to work before, the main difference is we're now using model.getContent() to retrieve the current notebook contents. I did a comparison of the contents that are being hashed in a debugger and they look the same, so could be an encoding issue? End result is that notebooks are trusted with one hash but checked for trust with a different hash, i.e. user-trusted notebooks are always untrusted on next load

ETA: model.getContent() returns stringified notebook contents that differ from what's actually stored on disk. This is the root cause of the problem. A proper fix for this would ensure that getContent() returns data that is identical to what's on disk

ETA to the ETA: There are actually two separate problems. The first is that getContent() sometimes unnecessarily reorders the JSON. The second is that JSON.parse converts escaped unicode characters to actual unicode, so the constructed model will differ from the actual file contents on disk until the user hits save

/cc @DonJayamanne

[joyceerhl]

Yeah it's an encoding issue:

[joyceerhl]

@rchiodo, Don can't repro this but he's on a mac. Any chance you can repro since you're on Windows?

[rchiodo]

Do you have a sample notebook?

[joyceerhl]

The one I can repro this with is here: https://github.com/Azure/MachineLearningNotebooks/blob/master/how-to-use-azureml/reinforcement-learning/atari-on-distributed-compute/pong_rllib.ipynb

[joyceerhl]

Repro steps:

1. Open an untrusted notebook that contains some invalid unicode characters with autosave off
2. Click trust in the trust prompt
3. Close notebook without making further edits or saving
4. Reopen notebook, notebook is loaded as untrusted again

Root cause:
The digest for isNotebookTrusted is computed with the raw notebook contents, and these invalid unicode characters are escaped in the raw file, but the digest for trustNotebook is computed with model.getContent() which parses the invalid unicode characters as garbage characters, resulting in two unequal hashes. If, after step 2 in the repro steps, you hit save, we overwrite the file with our parsed contents, so trust is correctly persisted and the notebook will be loaded trusted the next time. Both @kimadeline and @claudiaregio have run into this issue while testing, so I think it's common enough to be worth fixing even though the root cause is invalid unicode characters in the notebook

[kimadeline]

Is it what causes the second half of https://github.com/microsoft/vscode-python/issues/12867?

[joyceerhl]

@kimadeline I'm pretty sure it is, would you mind sending me the ipynb you used for that gif so I can try to repro?

[joyceerhl]

Looks like we're gonna run into this issue as long as we use model.getContent() for trustNotebooks and something else for checking if a notebook is trusted. Unicode isn't the problem per se. In the screenshot below, right pane is the notebook contents used when checking if the notebook @kimadeline provided is trusted on load, and left pane is the notebook contents from calling model.getContent(). IMO we need to consistently use either the raw file contents or model.getContent(), otherwise we risk this happening all over the place

[rchiodo]

Validated. Works for me on the sample notebook.