Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pageload experiment: -O2, -Oz build flags #7795

Open
1 of 4 tasks
liuche opened this issue Jan 18, 2020 · 13 comments
Open
1 of 4 tasks

Pageload experiment: -O2, -Oz build flags #7795

liuche opened this issue Jan 18, 2020 · 13 comments
Labels
perf:pageload performance pin

Comments

@liuche
Copy link
Contributor

liuche commented Jan 18, 2020

edit: see #7795 (comment) for what the next steps for this issue are.

Why/User Benefit/User Problem

Switching to the -O2 flag has pageload perf improvements, but increases apk size by ~6MB.

Need this experiment to measure tradeoffs.

What/Requirements

  • Stability experimentation on 3.2 1/22-1/27 w/ GV 74 Nightly
  • A/B test on Firefox Preview Release (pending Phase 1): 1/27-2/14 w/ GV 73 Beta
  • Decide which version of GV to use for upcoming releases
  • Update GV version in AC right before 3.3 goes out

This experiment would be run in staged rollout through Google Play, because Mako doesn't have this capability. Johan is assisting with this.
Corey from the Data Science team is involved in analyzing the results for retention.

Acceptance Criteria (how do I know when I’m done?)

Data Science has results that allow us to make a perf decision.

Meeting notes

┆Issue is synchronized with this Jira Task

@liuche liuche added the performance label Jan 18, 2020
@liuche liuche self-assigned this Jan 18, 2020
@liuche liuche added this to In Progress in Fenix Sprint Kanban Jan 18, 2020
@github-actions github-actions bot added the needs:triage label Jan 18, 2020
@yoasif
Copy link
Contributor

yoasif commented Jan 18, 2020

Don't want to spam here, but I think this is a no-brainer - if it actually results in pageload performance, a larger apk size pays for itself in terms of marketshare and people willing to try the browser because of the page load performance.

@liuche liuche removed the needs:triage label Jan 18, 2020
@mcomella mcomella added this to Low impact (unprioritized) in Performance, front-end roadmap Jan 21, 2020
@liuche liuche mentioned this issue Jan 23, 2020
32 tasks
@liuche
Copy link
Contributor Author

liuche commented Jan 25, 2020

@liuche
Copy link
Contributor Author

liuche commented Feb 1, 2020

@ecsmyth
Copy link

ecsmyth commented Feb 14, 2020

Updated pre-experimental impact analysis. Initial phase 2 experiment results expected o/a Feb 18

@ecsmyth
Copy link

ecsmyth commented Feb 19, 2020

Initial analysis of phase 2 data found no statistically significant difference in crash rates and a statistically significant improvement in page load performance.

@mcomella mcomella moved this from Waiting to Done in Performance, front-end roadmap Feb 21, 2020
@mcomella mcomella moved this from Done to Waiting in Performance, front-end roadmap Feb 21, 2020
@mcomella mcomella moved this from Waiting to In progress in Performance, front-end roadmap Feb 21, 2020
@mcomella mcomella assigned liuche and ecsmyth and unassigned liuche Feb 21, 2020
@ecsmyth
Copy link

ecsmyth commented Feb 25, 2020

Plan for phase 3

@ecsmyth ecsmyth moved this from In progress to Waiting in Performance, front-end roadmap Feb 25, 2020
@liuche
Copy link
Contributor Author

liuche commented Mar 24, 2020

Update here: We'll be rolling out 4.2 with GV Beta 75, which does not have the experiment enabled.

@liuche liuche removed this from In Progress in Fenix Sprint Kanban May 7, 2020
@ecsmyth ecsmyth moved this from Waiting to In progress in Performance, front-end roadmap Jun 24, 2020
@ecsmyth ecsmyth moved this from In progress to Top 10 Inter-Team bugs in Performance, front-end roadmap Aug 5, 2020
@stale
Copy link

stale bot commented Jun 15, 2021

See: #17373 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Jun 15, 2021
@lnicola
Copy link

lnicola commented Jun 15, 2021

This still sounds like it might be worth doing?

@stale stale bot removed the wontfix label Jun 15, 2021
@acreskeyMoz
Copy link
Collaborator

acreskeyMoz commented Jun 15, 2021

We didn't find enough evidence of performance improvement in the experiment to justify the increase in APK size.

That being said, this strategy, choosing build optimization that favors speed, still has future potential.
• We may be able to better measure performance in the wild at a future date
• We may be able to reduce the APK size and thus "pay for" the increase in size that this change leads to
• There is another build option -Os, that's also faster than what we use now, but leads to a smaller increase in APK size that -O2

@lnicola
Copy link

lnicola commented Jun 15, 2021

Ah, I just read through the discussion in https://bugzilla.mozilla.org/show_bug.cgi?id=1670948. I still think the measured improvements like in https://bugzilla.mozilla.org/show_bug.cgi?id=1591725#c46 are more relevant than the A/B testing results (if Firefox got 2x faster, most the I know wouldn't use it more), but at least it was tried.

@acreskeyMoz
Copy link
Collaborator

acreskeyMoz commented Jun 15, 2021

Yes, I'm also in favor of exploring this change again.
I just brought it up with my manager to consider for future mobile projects.

@stale
Copy link

stale bot commented Dec 12, 2021

See: #17373 This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the wontfix label Dec 12, 2021
@MarcLeclair MarcLeclair added pin and removed wontfix labels Dec 15, 2021
@mcomella mcomella moved this from Perf P1 (unordered) to Triaged (unordered) in Performance, front-end roadmap Jan 19, 2022
@mcomella mcomella added the perf:pageload label Jun 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
perf:pageload performance pin
Projects
Development

No branches or pull requests

7 participants