[HOLD for payment 2023-06-23] [$1000] valid html code displays 'this is beginning...' and invalid html code displays weird symbol in LHN for few seconds

[kavimuru]

If you haven’t already, check out our contributing guidelines for onboarding and email contributors@expensify.com to request to join our Slack channel!

---

Action Performed:

1. Open the app
2. Open any report
3. Send valid html code. eg:  
4. Observe the message in LHN
5. Send invalid html code. eg: 
6. Observe the message in LHN

Expected Result:

App should display the message as it is in LHN

Actual Result:

App displays 'This is beginning...' message for valid HTML code and weird emoji for invalid HTML code for few seconds in LHN

Workaround:

Can the user still use Expensify without this being fixed? Have you informed them of the workaround?

Platforms:

Which of our officially supported platforms is this issue occurring on?

• Android / native
• Android / Chrome
• iOS / native
• iOS / Safari
• MacOS / Chrome / Safari
• MacOS / Desktop

Version Number: 1.3.19-6
Reproducible in staging?: y
Reproducible in production?: y
If this was caught during regression testing, add the test name, ID and link from TestRail:
Email or phone of affected tester (no customers):
Logs: https://stackoverflow.com/c/expensify/questions/4856
Notes/Photos/Videos: Any additional supporting documentation

valid.invalid.html.code.lhn.issue.mp4

Recording.804.mp4

Expensify/Expensify Issue URL:
Issue reported by: @dhanashree-sawant
Slack conversation: https://expensify.slack.com/archives/C049HHMV9SM/p1685097977437509

View all open jobs on GitHub

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~0114e290337f545c10
• Upwork Job ID: 1663626267506884608
• Last Price Increase: 2023-05-30

[melvin-bot]

Triggered auto assignment to @stephanieelliott (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[melvin-bot]

Bug0 Triage Checklist (Main S/O)

• This "bug" occurs on a supported platform (ensure Platforms in OP are ✅)
• This bug is not a duplicate report (check E/App issues and #expensify-bugs)
  • If it is, comment with a link to the original report, close the issue and add any novel details to the original issue instead
• This bug is reproducible using the reproduction steps in the OP. S/O
  • If the reproduction steps are clear and you're unable to reproduce the bug, check with the reporter and QA first, then close the issue.
  • If the reproduction steps aren't clear and you determine the correct steps, please update the OP.
• This issue is filled out as thoroughly and clearly as possible
  • Pay special attention to the title, results, platforms where the bug occurs, and if the bug happens on staging/production.
• I have reviewed and subscribed to the linked Slack conversation to ensure Slack/Github stay in sync

[dukenv0307]

Proposal

Please re-state the problem that we are trying to solve in this issue.

valid html code displays 'this is beginning...' and invalid html code displays weird symbol in LHN for few seconds

What is the root cause of that problem?

optimisticReportData has lastMessageText is lastCommentText that is result of formatReportLastMessageText function

App/src/libs/actions/Report.js

Line 213 in 06192ac

const lastCommentText = ReportUtils.formatReportLastMessageText(lastAction.message[0].text);

In formatReportLastMessageText function, we use Str.htmlDecode and trim() to format. With html code  , it is space so that formatReportLastMessageText function returns '' and LHN displays "This is beginning....". With other valid html code, it display decode of that for few second by optimisticReportData, but when API returns data that contains lastMessageText is the correct message user entered instead of html decode of that, that makes LHN displays inconsistent

App/src/libs/ReportUtils.js

Line 643 in 06192ac

return Str.htmlDecode(String(lastMessageText)).replace(CONST.REGEX.AFTER_FIRST_LINE_BREAK, '').substring(0, CONST.REPORT.LAST_MESSAGE_TEXT_MAX_LENGTH).trim();

What changes do you think we should make in order to solve the problem?

We should remove Str.htmlDecode in formatReportLastMessageText to be consistent with data from BE and not display "This is beginning..." in LHN for some valid html like  

App/src/libs/ReportUtils.js

Line 643 in 06192ac

return Str.htmlDecode(String(lastMessageText)).replace(CONST.REGEX.AFTER_FIRST_LINE_BREAK, '').substring(0, CONST.REPORT.LAST_MESSAGE_TEXT_MAX_LENGTH).trim();

What alternative solutions did you explore? (Optional)

[s-alves10]

Proposal

Please re-state the problem that we are trying to solve in this issue.

When you send valid html code like  , LHN displays this is beginning ....
When you send invalid html like `�1, LHN displays weird symbol.

What is the root cause of that problem?

The root cause of this issue is in the following function
https://github.com/s-alves10/App/blob/95efb234dc6d9c14d01fa7ffafb2c3d90bfd82f7/src/libs/ReportUtils.js#L642-L644

We decode the lastMessageText first and add some processing to it. But lastMessageText was already decoded text as you can see from where the above function is used.

Double decoding is the main reason.

What changes do you think we should make in order to solve the problem?

Simply removing the decoding part in the function would solve the issue.

function formatReportLastMessageText(lastMessageText) {
    return String(lastMessageText).replace(CONST.REGEX.AFTER_FIRST_LINE_BREAK, '').substring(0, CONST.REPORT.LAST_MESSAGE_TEXT_MAX_LENGTH).trim();
}

This works perfectly.

This issue is similar to issue 19539. Both issues can be solved in a PR

Result

mac_chrome_19789.1.mp4

What alternative solutions did you explore? (Optional)

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~0114e290337f545c10

[melvin-bot]

Current assignee @stephanieelliott is eligible for the External assigner, not assigning anyone new.

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @parasharrajat (External)

[melvin-bot]

Triggered auto assignment to @stitesExpensify (External), see https://stackoverflow.com/c/expensify/questions/7972 for more details.

[stephanieelliott]

Hey @parasharrajat looks like there are a few proposals already in the scrollback.

[therealsujitk]

Very similar to #19539. If we can get the logic that the server is using we can prevent similar issues from coming up in the future.

[parasharrajat]

I see. We can wait on that issue to be completed and see if this is fixed as well.

[stitesExpensify]

Yes I think we should wait. I'll update the title

[therealsujitk]

After some testing I found that the issue cannot be resolved by simply removing Str.htmlDecode() - The reason.

The reason for this issue is with the Expensify/expensify-common package and this is the line that's causing it.

https://github.com/Expensify/expensify-common/blob/c898563fe851d9a4d594fa9afbdd1ddab5971636/lib/ExpensiMark.js#L589

underscore's unescape() function only decodes a limited set of characters (& becomes &, but it doesn't decode html entities such as  ).

Because of this only few parts of the string are being unescaped before it is returned. To resolve this _.unescape(replacedText); will have to be replaced with Str.htmlDecode(replacedText);.

Slack Discussion - https://expensify.slack.com/archives/C01GTK53T8Q/p1685590681879659

---

The PR I created (#19962) fixes #19539 but not this one. My suggestion is to fix the issue with this package first and then create a PR to update the dependency and fix issues such as this one, doing this would also prevent any new bugs or regressions.

I found a few instances of Str.htmlDecode() being used on top of parser.htmlToText(), all of them will have to be removed.

[therealsujitk]

I've opened a PR for this - Expensify/expensify-common#543. Also, I believe this issue is a regression from #19237.

[therealsujitk]

Proposal

Please re-state the problem that we are trying to solve in this issue.

When a user types in HTML entities such as  , they get rendered.

What is the root cause of that problem?

The reason for this is because we're unescaping the text string twice, first using _.unescape() in Expensify/expensify-common and second using Str.htmlDecode() in this repository.

What changes do you think we should make in order to solve the problem?

To resolve this we switch to using Str.htmlDecode() in Expensify/expensify-common and remove the use of Str.htmlDecode() in this repository when it's used on top of parser.htmlToText().

The reason we have to use Str.htmlDecode() is because _.unescape() decodes only a limited set of entities and leaves out the rest (ex:  ) resulting in an incomplete decode. This will fix issues such as this. Why this is necessary is further explained here.

What alternative solutions did you explore? (Optional)

None.

[therealsujitk]

I've added my proposal because I haven't been assigned to this issue, sorry for creating a PR in Expensify/expensify-common without being assigned. I'll close the PR if we decide to go with another proposal.

[melvin-bot]

Based on my calculations, the pull request did not get merged within 3 working days of assignment. Please, check out my computations here:

• when @alitoshmatov got assigned: 2023-06-07 15:00:26 Z
• when the PR got merged: 2023-06-14 14:51:56 UTC
• days elapsed: 5

On to the next one 🚀

[melvin-bot]

Triggered auto assignment to @Christinadobrzyn (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details.

[stephanieelliott]

Reapplying the Bug label to get another BZ member on this while I am OOO til July 3. Thanks @Christinadobrzyn, this PR was deployed to staging yesterday so probably not much to do here other than wait to issue payment!

[parasharrajat]

Note: This is a regression and the only Pending payment here will be reporting bonus.

[Christinadobrzyn]

moving to weekly since PR is under review

[s77rt]

No PR is under review. #20443 is merged. Waiting to be deployed to production

[melvin-bot]

Reviewing label has been removed, please complete the "BugZero Checklist".

[melvin-bot]

The solution for this issue has been 🚀 deployed to production 🚀 in version 1.3.28-5 and is now subject to a 7-day regression period 📆. Here is the list of pull requests that resolve this issue:

• Removed redundant Str.htmlDecode #20443

If no regressions arise, payment will be issued on 2023-06-23. 🎊

After the hold period is over and BZ checklist items are completed, please complete any of the applicable payments for this issue, and check them off once done.

• External issue reporter
• Contributor that fixed the issue
• Contributor+ that helped on the issue and/or PR

As a reminder, here are the bonuses/penalties that should be applied for any External issue:

• Merged PR within 3 business days of assignment - 50% bonus
• Merged PR more than 9 business days after assignment - 50% penalty

[melvin-bot]

BugZero Checklist: The PR fixing this issue has been merged! The following checklist (instructions) will need to be completed before the issue can be closed:

• [@s77rt] The PR that introduced the bug has been identified. Link to the PR:
• [@s77rt] The offending PR has been commented on, pointing out the bug it caused and why, so the author and reviewers can learn from the mistake. Link to comment:
• [@s77rt] A discussion in #expensify-bugs has been started about whether any other steps should be taken (e.g. updating the PR review checklist) in order to catch this type of bug sooner. Link to discussion:
• [@s77rt] Determine if we should create a regression test for this bug.
• [@s77rt] If we decide to create a regression test for the bug, please propose the regression test steps to ensure the same bug will not reach production again.
• [@stephanieelliott / @Christinadobrzyn] Link the GH issue for creating/updating the regression test once above steps have been agreed upon:

[s77rt]

• The PR that introduced the bug has been identified: Last visible message text, expensify-common update #19237
• The offending PR has been commented on: Last visible message text, expensify-common update #19237 (comment)
• A discussion in #expensify-bugs has been started: I'm pretty sure we don't have much to discuss here, the encoding issue was a thing going for a long time and we started fixing this a couple of weeks ago, this is just a propagation of the fix.
• Determine if we should create a regression test for this bug: No

[Christinadobrzyn]

Hired in Upwork

Internal job: https://www.upwork.com/ab/applicants/1663626267506884608/job-details
External job: https://www.upwork.com/jobs/~0114e290337f545c10

@dhanashree-sawant reporter
@parasharrajat C+
@alitoshmatov contributor

Speed bonus applies - #19789 (comment)

[parasharrajat]

@Christinadobrzyn There will only be reporting payment here. This was a regression issue and handled by the original author and C+ so no payment for C+ and Contributor.

#19789 (comment)

[Christinadobrzyn]

Ah thanks @parasharrajat! I'll just pay the reporter when we're ready!

[Christinadobrzyn]

Paid @dhanashree-sawant for reporting. Closed job in Upwork. Closing this GH.