GmailElementReplacer refactoring #5069

rrrooommmaaa · 2023-04-05T17:08:02Z

This PR is WIP

close #5022
close #3944
close #4164
close #4810

Tests (delete all except exactly one):

Tests added or updated

To be filled by reviewers

I have reviewed that this PR... (tick whichever items you personally focused on during this review):

addresses the issue it closes (if any)
code is readable and understandable
is accompanied with tests, or tests are not needed
is free of vulnerabilities
is documented clearly and usefully, or doesn't need documentation

…ement-replacer-refactor-live-test

extension/js/content_scripts/webmail/gmail-element-replacer.ts

…ement-replacer-refactor-live-test

…-live-test

…ement-replacer-refactor-live-test

extension/chrome/elements/pgp_render_block.ts

…ement-replacer-refactor-live-test

extension/js/content_scripts/webmail/gmail-element-replacer.ts

…ement-replacer-refactor-live-test

…e message

extension/js/content_scripts/webmail/setup-webmail-content-script.ts

extension/js/common/relay-manager.ts

…-live-test

sosnovsky · 2023-06-19T12:59:26Z

@rrrooommmaaa I built extension and now can't open any encrypted message, got such errors:

Error: BrowserMsg(db) sendRawResponse::TypeError: Cannot read properties of undefined (reading 'readKeys')`

Could not open this message with FlowCrypt.
other: TypeCannot read properties of undefined (reading 'readMessage')

Error: BrowserMsg(db) sendRawResponse::TypeError: undefined has no properties

Could not open this message with FlowCrypt.
other: Typeopgp is undefined

sosnovsky · 2023-06-19T13:03:59Z

BrowserMsg(db) sendRawResponse::TypeError: undefined has no properties
BrowserMsg.jsonToErr@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:8805:15
processRawMsgResponse@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:8720:35


[callerStack]
BrowserMsg.sendAwait@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:8681:18
db@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:8435:36
ContactStore.getOneWithAllPubkeys@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:1478:47
MessageRenderer.getVerificationPubs@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:11067:44
MessageRenderer/this.renderCryptoMessage@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:10809:74


url: https://mail.google.com/mail/u/0/#inbox/FMfcgzGsmrCDZttTrQKbnKckXSXKZLzn

[/callerStack]

[responderStack]
Catch.rewrapErr@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/platform/catch.js:35:20
BrowserMsg.errToJson@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/browser/browser-msg.js:397:38
BrowserMsg.sendRawResponse/<@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/browser/browser-msg.js:418:67


[typeof:Error:TypeError] undefined has no properties

OpenPGPKey.parseMany@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/core/crypto/pgp/openpgp-key.js:23:24
KeyUtil.parseMany@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/core/crypto/key.js:99:33
KeyUtil.parse@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/core/crypto/key.js:89:32
ContactStore.sortKeys/pubkeyInfos<@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/platform/store/contact-store.js:716:34
ContactStore.sortKeys@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/platform/store/contact-store.js:715:51
ContactStore.getOneWithAllPubkeys@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/common/platform/store/contact-store.js:563:46

[/responderStack]

sosnovsky · 2023-06-19T13:05:02Z

webmail_bundle.js:68 Error: BrowserMsg(db) sendRawResponse::TypeError: Cannot read properties of undefined (reading 'readKeys')
    at BrowserMsg.jsonToErr (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:10002:15)
    at processRawMsgResponse (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:9908:39)

[callerStack]
    at chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:9877:30
    at new Promise (<anonymous>)
    at BrowserMsg.sendAwait (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:9869:22)
    at Object.db (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:9612:36)
    at ContactStore.getOneWithAllPubkeys (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:1549:51)
    at MessageRenderer.getVerificationPubs (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:12567:48)
    at MessageRenderer.renderCryptoMessage (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:12269:78)
    at async MessageRenderer.renderMsgBlock (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:12197:24)

url: https://mail.google.com/mail/u/0/#inbox/FMfcgzGsmDrQHdnkVcnWSFlhJnhfzJnD

[/callerStack]

[responderStack]
Error: sendRawResponse::TypeError: Cannot read properties of undefined (reading 'readKeys')
    at Catch.rewrapErr (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/platform/catch.js:35:20)
    at BrowserMsg.errToJson (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/browser/browser-msg.js:417:38)
    at chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/browser/browser-msg.js:438:67

[typeof:Error:TypeError] Cannot read properties of undefined (reading 'readKeys')

TypeError: Cannot read properties of undefined (reading 'readKeys')
    at OpenPGPKey.parseMany (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/core/crypto/pgp/openpgp-key.js:29:39)
    at KeyUtil.parseMany (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/core/crypto/key.js:120:37)
    at KeyUtil.parse (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/core/crypto/key.js:105:36)
    at chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/platform/store/contact-store.js:780:42
    at Array.map (<anonymous>)
    at ContactStore.sortKeys (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/platform/store/contact-store.js:778:55)
    at ContactStore.getOneWithAllPubkeys (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/common/platform/store/contact-store.js:594:50)
    at async BgHandlers.dbOperationHandler (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/background_page/bg-handlers.js:34:16)
    at <async> parseMany (/extension/js/common/core/crypto/pgp/openpgp-key.ts:38:28)
    at <async> parseMany (/extension/js/common/core/crypto/key.ts:191:28)
    at <async> parse (/extension/js/common/core/crypto/key.ts:182:24)
    at <async> <anonymous> (/extension/js/common/platform/store/contact-store.ts:768:19)
    at <async> sortKeys (/extension/js/common/platform/store/contact-store.ts:765:28)
    at <async> getOneWithAllPubkeys (/extension/js/common/platform/store/contact-store.ts:549:39)
    at <async> dbOperationHandler (/extension/js/background_page/bg-handlers.ts:17:37)
[/responderStack]

    at <async> sendAwait (/extension/js/common/browser/browser-msg.ts:456:29)
    at <async> getOneWithAllPubkeys (/extension/js/common/platform/store/contact-store.ts:549:39)
    at <async> getVerificationPubs (/extension/js/common/message-renderer.ts:215:39)
    at <async> renderCryptoMessage (/extension/js/common/message-renderer.ts:686:32)
    at <async> renderMsgBlock (/extension/js/common/message-renderer.ts:613:27)

### Catch.reportErr calling stack ###
#     at Catch.formatExceptionForReport (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:277:90)
#     at Catch.onErrorInternalHandler (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:85:29)
#     at Catch.reportErr (chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:101:18)
#     at chrome-extension://pjohdcpiapjhagmakkjpgjmodmgfmkkd/js/content_scripts/webmail_bundle.js:12185:27
# 
# url: https://mail.google.com/mail/u/0/#inbox/FMfcgzGsmDrQHdnkVcnWSFlhJnhfzJnD
# 
######################

rrrooommmaaa · 2023-06-19T13:31:22Z

Could not open this message with FlowCrypt.
other: Typeopgp is undefined

What browser is it?
Have you re-built in full? (with npm install)?

sosnovsky · 2023-06-19T13:37:33Z

Could not open this message with FlowCrypt.
other: Typeopgp is undefined

This error happened in Firefox.
I tried in both - Chrome and Firefox, there were different errors for different messages.
You can check attached logs from console - #5069 (comment) from Firefox, #5069 (comment) from Chrome

What browser is it? Have you re-built in full? (with npm install)?

Yes, I ran npm install and then also tried to re-build with npm run build

sosnovsky · 2023-06-19T13:41:45Z

I checked console and there is error even without opening any email, just after loading gmail page:

SyntaxError: expected expression, got keyword 'return' [openpgp.js:4378:7](moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/lib/openpgp.js)
Loading FlowCrypt 8.4.8 [webmail_bundle.js:13150:29](moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js)
Error: Some keys could not be parsed
    parseAndCheckPrivateKeys moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:10025
    # Catch.onErrorInternalHandler moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:85
    # Catch.reportErr moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:101
    # processKeysFromEkm moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:13364
[webmail_bundle.js:68:13](moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js)
Some keys could not be parsed
parseAndCheckPrivateKeys@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:10025:19
[webmail_bundle.js:72:13](moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js)
TypeError: opgp is undefined
    parseMany moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:4561
    parseMany moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:3903
    parse moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:3893
    expirations moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:10126
    getLocalKeyExpiration moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:10126
    # Catch.onErrorInternalHandler moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:85
    # Catch.reportErr moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:101
[webmail_bundle.js:68:13](moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js)
opgp is undefined
OpenPGPKey.parseMany@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:4561:24
KeyUtil.parseMany@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:3903:33
KeyUtil.parse@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:3893:32
getLocalKeyExpiration/expirations<@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:10126:80
getLocalKeyExpiration@moz-extension://cc209f92-c631-4136-896d-fd5c240338aa/js/content_scripts/webmail_bundle.js:10126:47

sosnovsky · 2023-06-19T13:43:36Z

Line with error in openpgp.js

4377  function is_bytes(a) {
4378 (     return a instanceof Uint8Array ||      return a instanceof globalThis.Uint8Array);
4379  }

rrrooommmaaa · 2023-06-19T13:47:33Z

I tried in both - Chrome and Firefox, there were different errors for different messages.

So these are both on Mac?

function is_bytes(a)

I see something wrong with regexes again...

sosnovsky · 2023-06-19T13:58:30Z

So these are both on Mac?

Yes, that's on Mac, but there wasn't any errors during build. I can re-check later today which regexes from build script may fail on Mac

rrrooommmaaa · 2023-06-19T14:16:14Z

That's because of greediness.

Should be return (a instanceof Uint8Array || a instanceof globalThis.Uint8Array); instead of ( return a instanceof Uint8Array || return a instanceof globalThis.Uint8Array);

…-live-test

rrrooommmaaa · 2023-06-19T14:28:22Z

That's because of greediness.

I pushed the lazy-matching fix. Can you build it again on Mac?

sosnovsky · 2023-06-19T18:01:39Z

That's because of greediness.

I pushed the lazy-matching fix. Can you build it again on Mac?

No, currently I get error:

+ backup_ext=''\'''\'''
+ for file in "$@"
+ sed '-i'\'''\''' -E 's/\(([^\(\)\x20]+?)\x20instanceof\x20Uint8Array\)/\(\1\x20instanceof\x20Uint8Array\x20\|\|\x20\1\x20instanceof\x20globalThis\.Uint8Array\)/g' ./build/generic-extension-wip/lib/openpgp.js
sed: 1: "s/\(([^\(\)\x20]+?)\x20 ...": RE error: repetition-operator operand invalid

…ement-replacer-refactor-live-test

rrrooommmaaa · 2023-06-20T12:19:06Z

This means the lazy modificator isn't supported. I guess, /U isn't supported either.
I'm a bit confused by this behavior as I specified the space character to be excluded with [^\(\)\x20].
How could return a have been captured?
We can try to replace \x20 with \s if it helps.
But wouldn't be better to use a package with the same behavior on all platforms,
some package installable with npm? e.g. https://www.npmjs.com/package/replace

sosnovsky · 2023-06-20T12:53:02Z

I updated ISUINT8ARRAY_REGEX3 to be

ISUINT8ARRAY_REGEX3="s/return\x20([^\(\)\x20]+)\x20instanceof\x20Uint8Array;/return\x20\(\1\x20instanceof\x20Uint8Array\x20\|\|\x20\1\x20instanceof\x20globalThis\.Uint8Array\);/g"

and it worked well, all occurrences in openpgp.js were replaced. Maybe let's just use this regex?

rrrooommmaaa · 2023-06-20T13:04:32Z

Maybe let's just use this regex?

Yes, we can make do with this regex.

sosnovsky

Great job, I haven't found any issues - everything works well. Maybe I missed something, as list of changes is very big, but if anything will be found - it can be fixed in another PR, this one can be merged so there won't be merge conflicts with other PRs.

Roman Shevchenko added 5 commits April 5, 2023 18:06

(wip) move in the direction of common renderer

f5e09bc

fix

73082cf

fix

e91bc1b

(wip) better attachment recognition

928fbbf

Merge remote-tracking branch 'origin/master' into issue-5022-gmail-el…

db8e0ba

…ement-replacer-refactor-live-test

github-advanced-security bot found potential problems Apr 12, 2023

View reviewed changes

extension/js/content_scripts/webmail/gmail-element-replacer.ts Fixed Show fixed Hide fixed

extension/js/content_scripts/webmail/gmail-element-replacer.ts Fixed Show fixed Hide fixed

Roman Shevchenko and others added 14 commits April 12, 2023 08:41

Merge remote-tracking branch 'origin/master' into issue-5022-gmail-el…

e80fb02

…ement-replacer-refactor-live-test

lint fix

38ee474

extended test for #5029 for GmailElementReplacer

fad9db3

fix signature attachment

02d31d8

Merge remote-tracking branch 'origin/master' into issue-5022-gmail-el…

bacb7a3

…ement-replacer-refactor-live-test

Merge remote-tracking branch 'origin/master' into issue-5022-gmail-el…

6966668

…ement-replacer-refactor-live-test

Temporarily increase test overall timeout

269b55f

fix

2e1101b

fix

598735d

inrease execution timeout

557ea8d

Merge remote-tracking branch 'origin/master' into issue-5022-gmail-el…

c7a6419

…ement-replacer-refactor-live-test

Merge branch 'master' into issue-5022-gmail-element-replacer-refactor…

71ec76d

…-live-test

Merge remote-tracking branch 'origin/master' into issue-5022-gmail-el…

e7cd1ed

…ement-replacer-refactor-live-test

(wip) render-only pgp block iframe

f3313fb

github-advanced-security bot found potential problems Apr 20, 2023

View reviewed changes

extension/chrome/elements/pgp_render_block.ts Fixed Show fixed Hide fixed

Merge remote-tracking branch 'origin/master' into issue-5022-gmail-el…

88f1713

…ement-replacer-refactor-live-test

rrrooommmaaa commented Apr 21, 2023

View reviewed changes

extension/js/content_scripts/webmail/gmail-element-replacer.ts Outdated Show resolved Hide resolved

Roman Shevchenko added 3 commits April 21, 2023 10:53

Merge remote-tracking branch 'origin/master' into issue-5022-gmail-el…

719637a

…ement-replacer-refactor-live-test

test fix

c6ffa6d

fixed relay renderer to await for the destination frame readyToReceiv…

43219bf

…e message

github-advanced-security bot found potential problems Apr 22, 2023

View reviewed changes

extension/js/content_scripts/webmail/setup-webmail-content-script.ts Fixed Show fixed Hide fixed

extension/js/common/relay-manager.ts Fixed Show fixed Hide fixed

Roman Shevchenko added 3 commits April 22, 2023 14:52

update test and recipe to check pgp block for correctness

21d8d18

show signature badge

4f881d9

fix test recipe

ab682c9

rrrooommmaaa requested a review from sosnovsky June 19, 2023 08:50

rrrooommmaaa added 3 commits June 19, 2023 12:54

Merge branch 'master' into issue-5022-gmail-element-replacer-refactor…

ee4ff6e

…-live-test

Merge branch 'master' into issue-5022-gmail-element-replacer-refactor…

c10350c

…-live-test

Merge branch 'master' into issue-5022-gmail-element-replacer-refactor…

8797e02

…-live-test

fix script

1d98bc2

Roman Shevchenko and others added 2 commits June 19, 2023 15:23

lazy matching

4a2fc81

Merge branch 'master' into issue-5022-gmail-element-replacer-refactor…

63fd738

…-live-test

Merge remote-tracking branch 'origin/master' into issue-5022-gmail-el…

f036a00

…ement-replacer-refactor-live-test

sosnovsky mentioned this pull request Jun 20, 2023

Replace "test strategies" with direct message testing #5241

Open

fix build script on macOS

78613f9

sosnovsky approved these changes Jun 20, 2023

View reviewed changes

sosnovsky enabled auto-merge (squash) June 20, 2023 20:29

sosnovsky merged commit a5f53bc into master Jun 20, 2023

sosnovsky deleted the issue-5022-gmail-element-replacer-refactor-live-test branch June 20, 2023 20:56

rrrooommmaaa mentioned this pull request Jun 26, 2023

Finish settings/inbox page refactoring #5256

Open

GmailElementReplacer refactoring #5069

GmailElementReplacer refactoring #5069

Uh oh!

Conversation

rrrooommmaaa commented Apr 5, 2023 • edited by sosnovsky Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

close #5022 close #3944 close #4164 close #4810

To be filled by reviewers

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sosnovsky commented Jun 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sosnovsky commented Jun 19, 2023

Uh oh!

sosnovsky commented Jun 19, 2023

Uh oh!

rrrooommmaaa commented Jun 19, 2023

Uh oh!

sosnovsky commented Jun 19, 2023

Uh oh!

sosnovsky commented Jun 19, 2023

Uh oh!

sosnovsky commented Jun 19, 2023

Uh oh!

rrrooommmaaa commented Jun 19, 2023

Uh oh!

sosnovsky commented Jun 19, 2023

Uh oh!

rrrooommmaaa commented Jun 19, 2023

Uh oh!

rrrooommmaaa commented Jun 19, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sosnovsky commented Jun 19, 2023

Uh oh!

rrrooommmaaa commented Jun 20, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sosnovsky commented Jun 20, 2023

Uh oh!

rrrooommmaaa commented Jun 20, 2023

Uh oh!

sosnovsky left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

rrrooommmaaa commented Apr 5, 2023 •

edited by sosnovsky

Loading

close #5022
close #3944
close #4164
close #4810

sosnovsky commented Jun 19, 2023 •

edited

Loading

rrrooommmaaa commented Jun 19, 2023 •

edited

Loading

rrrooommmaaa commented Jun 20, 2023 •

edited

Loading