LOW: [$250] Stop using reportAction.originalMessage or reportAction.message.text

[quinthar]

Problem:

Each comment in the server database is stored in reportAction.message in HTML form. However, for some reason lost to time, when we send that same comment out as an Onyx update, we:

1. Rename the original message in the database to reportAction.originalMessage in the update (which is confusing)
2. Create a new message object that contains a copy of the html message, as well as a stripped text version.
3. Put the message object inside a totally unnecessary one-element array

That means every comment is actually sent out three times (four, if you include the update to the report object, but that's out of scope for now). This is clearly wasteful in a number of ways:

1. It requires postprocessing on the server to do this extraneous operation
2. It increases network bandwidth by 3x
3. It increases RAM use 3x
4. It increases on-device storage 3x
5. It's confusing AF

Additionally, it's particularly problematic when sending Onyx updates out via UrbanAirship, which is extremely limited in the payload sizes allowed -- with anything over a certain limit just dropped quietly and never delivered, causing "gaps" in our update stream (which require more network calls to "backfill" the Onyx data on app open, which makes things slow). Basically, this was just a mistake introduced for reasons we can't remember, but that we want to undo.

Solution:

To solve this, please:

1. Phase 1 (External / clientside): Stop using originalMessage, start using array-message
   1. Update every instance of originalMessage.html to use message.html || message[0].html
   2. Update every instance of message.text to dynamically strip the text from message.html || message[0].html
   3. Update every key within originalMessage to use the original message
2. Phase 2 (Internal / serverside): Once we've confirmed the client no longer uses originalMessage or message.text or any other key within originalMessage:
   1. Stop sending originalMessage
   2. Start sending message without an array
3. Phase 3 (External / clientside): Remove checking for "array message"

This issue is for phase 1 and phase 3.

Upwork Automation - Do Not Edit

• Upwork Job URL: https://www.upwork.com/jobs/~01856d1399295ef760
• Upwork Job ID: 1776729925881405440
• Last Price Increase: 2024-04-06
• Automatic offers:
• cubuspl42 | Reviewer | 0

[melvin-bot]

Job added to Upwork: https://www.upwork.com/jobs/~01856d1399295ef760

[melvin-bot]

Triggered auto assignment to Contributor-plus team member for initial proposal review - @cubuspl42 (External)

[maxim-grin]

Hey I’m going to investigate why the payload is so big in Onyx storage. Possibly track what change exactly caused the problem. Then either revert that piece of change or use one of suggestions by @thienlnam

Contributor details
Your Expensify account email: maxim.grin@gmail.com
Upwork Profile Link: https://www.upwork.com/freelancers/~01fb7b172c01252917

[melvin-bot]

📣 @maxim-grin! 📣
Hey, it seems we don’t have your contributor details yet! You'll only have to do this once, and this is how we'll hire you on Upwork.
Please follow these steps:

1. Make sure you've read and understood the contributing guidelines.
2. Get the email address used to login to your Expensify account. If you don't already have an Expensify account, create one here. If you have multiple accounts (e.g. one for testing), please use your main account email.
3. Get the link to your Upwork profile. It's necessary because we only pay via Upwork. You can access it by logging in, and then clicking on your name. It'll look like this. If you don't already have an account, sign up for one here.
4. Copy the format below and paste it in a comment on this issue. Replace the placeholder text with your actual details.
   
   Format:

Contributor details
Your Expensify account email: <REPLACE EMAIL HERE>
Upwork Profile Link: <REPLACE LINK HERE>

[maxim-grin]

Hey I’m going to investigate why the payload is so big in Onyx storage. Possibly track what change exactly caused the problem. Then either revert that piece of change or use one of suggestions by @thienlnam

Contributor details
Your Expensify account email: maxim.grin@gmail.com
Upwork Profile Link: https://www.upwork.com/freelancers/~01fb7b172c01252917

[melvin-bot]

✅ Contributor details stored successfully. Thank you for contributing to Expensify!

[ShridharGoel]

Proposal

Please re-state the problem that we are trying to solve in this issue.

We don't need to use text from reportAction, it would be better to use originalMessage.html.

What is the root cause of that problem?

New change.

text from message in reportAction is being used at multiple places.

What changes do you think we should make in order to solve the problem?

We have text defined here:

App/src/types/onyx/ReportAction.ts

Lines 18 to 19 in f150229

	/** The text content of the fragment. */
	text: string;

Example usage:

App/src/components/ReportActionItem/ReportPreview.tsx

Line 173 in f150229

const actionMessage = action.message?.[0]?.text ?? '';

We can change this to:

const parser = new ExpensiMark();
const actionMessage = parser.htmlToText(action.message?.[0]?.html ?? '');

Add the above in a common method getActionMessagePostParsing in ReportUtils.
Similarly, check all usages and update them to use the method mentioned above.

Finally, we can remove text.

There's one instance of originalMessage?.html in ReportUtils.ts which also needs to be updated to use the common method.

if (ReportActionsUtils.isWhisperAction(reportAction)) {
    // Allow flagging welcome message whispers as they can be set by any room creator
    if (report?.description && !isCurrentUserAction && isOriginalMessageHaveHtml && `getActionMessagePostParsing(reportAction)` === report.description) {
        return true;
    }

    // Disallow flagging the rest of whisper as they are sent by us
    return false;
}

[nkdengineer]

Proposal

Please re-state the problem that we are trying to solve in this issue.

We want to remove the usage of text of reportAction in the payload sent to client side.
And also changes how we're using message.html

What is the root cause of that problem?

This is new feature request.

What changes do you think we should make in order to solve the problem?

1. Define 2 common methods here, to get the report action html (with forward compatibility: message as single item), and to get the report action text dynamically by parsing the html:

function getReportActionHtml(reportAction: OnyxEntry<ReportAction>) {
    return reportAction?.message?.html || reportAction?.message?.[0]?.html;
}

function getReportActionText(reportAction: OnyxEntry<ReportAction>) {
    const html = getReportActionHtml(reportAction);
    const parser = new ExpensiMark();
    return html ? parser.htmlToText(html) : '';
}

2. Update every instance of originalMessage.html to use message.html || message[0].html

In here, replace reportAction?.originalMessage?.html by getReportActionHtml(reportAction)

3. Update every instance of message.text to dynamically strip the text from message.html || message[0].html

There're some places that are using message?.[0]?.text, examples: here, here, here, ... (There're more)
We can replace them by getReportActionText(reportAction).

4. Phase 3 (External / clientside): Remove checking for "array message"

After phase 2 is done, we can just change the implementation of getReportActionHtml to

function getReportActionHtml(reportAction: OnyxEntry<ReportAction>) {
    return reportAction?.message?.html;
}

5. There're other places that use originalMessage, but not its html field, like here

We might need to replace them with getReportActionMessage(reportAction)

Where getReportActionMessage will be the following:

function getReportActionMessage(reportAction: OnyxEntry<ReportAction>) {
    return reportAction?.message || reportAction?.message?.[0];
}

To be compatible with both current format of message (as array) and new message format later (single item). The reportAction?.message?.[0] part can be removed after Phase 2 is done.

What alternative solutions did you explore? (Optional)

NA

[cubuspl42]

@ShridharGoel

Similarly, check all usages and update them

Are you saying to literally repeat this code snippet in all places? Couldn't we refactor the code to avoid this repetition?

As the scope of the issue is well-defined and rather narrow, I believe this part of the plan can be a part of the proposal.

@nkdengineer

Aren't you quoting browser-specific local notifications code? 🤔 As I understand it, the ultimate goal is to optimize the payload size of remote push notifications.

[nkdengineer]

Aren't you quoting browser-specific local notifications code? 🤔 As I understand it, the ultimate goal is to optimize the payload size of remote push notifications.

@cubuspl42 In the OP it's mentioned

@thienlnam thinks we added text for notifications

So I'm quoting the place that we're showing notification based on text, so it's clear where it's used.

It's only 1 part of the proposal though, what do you think about the rest of my proposal?

Are you saying to literally repeat this code snippet in all places? Couldn't we refactor the code to avoid this repetition?

@cubuspl42 My alternative solution achieves this

[ShridharGoel]

Proposal

Updated. @cubuspl42 We can add a method in ReportUtils, updated the proposal.

[cubuspl42]

@quinthar

Would you summarize the Expensify notifications architecture in a few sentences so that we can be more confident with proposal preparation and review?

As I understand that most (if not all?) notification processing on mobile Native is server-side (push-based), and we're talking about optimizing the pushed payload size. But I'm not 100% confident how that relates to the Onyx schema.

Also, there were some doubts about the scope...

I'm assuming here that we're only omitting the text from reportActions in urban airship native updates, if we're going to remove text in Pusher update too, then we need to populate the text in the Pusher report actions updates by htmlToText.

Would you clarify this too? My intuition was that we wanted to nuke all usage of the text property client-side, also removing it from the typing. But it would be good to be sure about it.

[quinthar]

Great question. I've updated the original post with a more detailed description of the problem and solutions. Thanks!

[ShridharGoel]

Proposal

Updated

[quinthar]

Updated the proposal again (sorry!!!) in light of message being in this weird one-element array. Thanks for your understanding and help in cleaning up this mess!

[nkdengineer]

Proposal updated after latest requirements update

[cubuspl42]

@nkdengineer I think your proposal shows more preparation. How is your capacity, though? Would you be able to start working on this within 24-48 hours?

[nkdengineer]

Would you be able to start working on this within 24-48 hours?

@cubuspl42 Yes, I definitely can

[cubuspl42]

I approve the proposal by @nkdengineer

C+ reviewed 🎀 👀 🎀

[melvin-bot]

Triggered auto assignment to @thienlnam, see https://stackoverflow.com/c/expensify/questions/7972 for more details.

[melvin-bot]

Triggered auto assignment to Contributor Plus for review of internal employee PR - @parasharrajat (Internal)

[mallenexpensify]

@parasharrajat reassigning, please take over as C+. If you don't have bandwidth, unassign yourself. Thanks. PR is

• feature: Stop using reportAction.originalMessage or reportAction.messge #40168

[parasharrajat]

Sorry, I don't have the bandwidth to work on this. I was OOO since Thursday. let me request someone to push this forward.

[parasharrajat]

@c3024 will review this.

[mallenexpensify]

Thanks @parasharrajat , 👋 @c3024
@c3024 commented on the PR earlier today

[melvin-bot]

This issue has not been updated in over 15 days. @thienlnam, @c3024, @nkdengineer eroding to Monthly issue.

P.S. Is everyone reading this sure this is really a near-term priority? Be brave: if you disagree, go ahead and close it out. If someone disagrees, they'll reopen it, and if they don't: one less thing to do!

[mallenexpensify]

@c3024 looks like the PR is awaiting an action from you

• feature: Stop using reportAction.originalMessage or reportAction.messge #40168 (comment)

[c3024]

@thienlnam @quinthar

The issue talks about only text and html

Phase 1 (External / clientside): Stop using originalMessage, start using array-message
Update every instance of originalMessage.html to use message.html || message[0].html
Update every instance of message.text to dynamically strip the text from message.html || message[0].html

but originalMessage has many more fields like reason, resolution, IOUTransactionID, events, oldName, newName, participantAccountIDs, etc. which are not present in message. So this phase 1 should be to update every instance of originalMessage (not just fields of html or text) with array message

Also in phase 2

Phase 2 (Internal / serverside): Once we've confirmed the client no longer uses originalMessage or message.text:
Stop sending originalMessage
Start sending message without an array

message should be sent including all the above mentioned other fields to completely deprecate originalMessage.

Can you please update phase 1 and 2 of the issue description to include these?

[melvin-bot]

Triggered auto assignment to @puneetlath (Bug), see https://stackoverflow.com/c/expensify/questions/14418 for more details. Please add this bug to a GH project, as outlined in the SO.

[mallenexpensify]

Added Bug to assign a BZ. @quinthar @thienlnam , can you address @c3024 's request above when you have a min
#39797 (comment)

[thienlnam]

I've updated the description

[roryabraham]

I did some exploratory work on this topic in #41872 - maybe useful for you @nkdengineer @c3024 @thienlnam.

The main idea is that we make ReportAction take a generic param for the type, and that determines the originalMessage it will have. This helps ensure that we correctly and safely narrow the ReportAction type to ensure the originalMessage is as we expect before trying to access any of its properties

[roryabraham]

self-assigning as I've been pulled into the PR for review

[melvin-bot]

@puneetlath, @thienlnam, @roryabraham, @c3024, @nkdengineer Whoops! This issue is 2 days overdue. Let's get this updated quick!

[melvin-bot]

@puneetlath, @thienlnam, @roryabraham, @c3024, @nkdengineer Huh... This is 4 days overdue. Who can take care of this?