Reduce duplicates places queries #3695

piatra · 2017-10-12T01:25:41Z

Closes #3638 and fix #3697

Removed fetchHiglights call when there are less than 9 highlights cards
added an flag in Topsites and Highlights to signal that data is currently being fetched
- this will prevent INIT and NEW_TAB_LOAD actions from triggering two update calls
  - covers the case when the events are fast enough that they overlap
  - otherwise lastUpdated field will prevent requests from happening
on TOP_SITES_UPDATED check that Topsites and Highlights have items in common before triggering fetchHighlights
- fetchHiglights ensures deduping but we can skip it if there is nothing to dedupe

Mardak · 2017-10-12T03:30:20Z

This seems to revert the fix for #3377 from #3386

piatra · 2017-10-12T12:14:41Z

And probably something similar can happen for Highlights as well?
I'll do some more testing.

piatra · 2017-10-12T17:40:18Z

@Mardak I’ve simulated a delayed TopSitesFeed startup by doing await new Promise(resolve => setTimeout(resolve, 3000)); in the refresh method. On startup I also open several other new tabs. They are all missing TopSites initially but as soon as the first one is initialized they all get the data.

Does this flow match the one in the issue? I'm trying to reproduce it so that maybe we can fix it some other way that doesn't involve multiple calls to refresh.

Mardak · 2017-10-12T19:42:35Z

That's how things behaved before #3386. I just tracked the actual cause: NEW_TAB_LOAD happens before INIT, so the broadcast doesn't happen because that one gets skipped for a pending refresh from LOAD. Filed #3697

Mardak · 2017-10-12T19:57:46Z

@k88hudson I wonder if this is the right place/time to just get rid of refreshing sites/highlights on NEW_TAB_LOAD especially if both are refreshing based on last updated time. Probably do it on SYSTEM_TICK?

In terms of avoiding double Highlights places call on INIT and TOP_SITES_UPDATED, instead of awaiting for top sites from INIT, maybe just return early and remember that the next TOP_SITES_UPDATED should broadcast. Although if TopSitesFeed always broadcasts… should Highlights getting TOP_SITES_UPDATED also broadcast?

Mardak · 2017-10-12T19:59:08Z

Actually nevermind the last question. I would think SYSTEM_TICK does /not/ broadcast and just updates the store for the next rehydration. So similarly highlights shouldn't broadcast on TOP_SITES_UPDATED unless a previous broadcast was skipped because there was no top sites data.

piatra · 2017-10-12T21:32:22Z

I wonder if this is the right place/time to just get rid of refreshing sites/highlights on NEW_TAB_LOAD especially if both are refreshing based on last updated time. Probably do it on SYSTEM_TICK?

This would fix the extra calls on startup for both TopSites and Highlights.

In terms of avoiding double Highlights places call on INIT and TOP_SITES_UPDATED, instead of awaiting for top sites from INIT, maybe just return early and remember that the next TOP_SITES_UPDATED should broadcast.

Highlights should return early if Topsites are not ready and broadcast if its store state is not initialized (this should only happen for initialization).

Mardak · 2017-10-12T21:47:11Z

Oh, checking for Highlights's section initialized value should work. I just realized Highlights probably shouldn't refresh on any actions that TopSitesFeed is also watching; otherwise, we end up with the double updates as we do now when TOP_SITES_UPDATED comes in. Although I can't think of a good way to maintain that dependency right now…

Or maybe something like:

constructor() {
  this.broadcastNextUpdate = true;
}
onAction() {
  case MIGRATION:
  case PLACES_*:
    this.broadcastNextUpdate = true;
    break;
  case TOP_SITES_UPDATED:
    this.fetchHighlights();
}

k88hudson · 2017-10-13T14:43:28Z

system-addon/lib/HighlightsFeed.jsm

@@ -54,6 +55,12 @@ this.HighlightsFeed = class HighlightsFeed {
  }

  async fetchHighlights(broadcast = false) {
+    if (this._updateInProgress) {


The only issue I've seen with this approach is if an update is in progress where an actual change happened (e.g. something was edited, blocked) the old data is returned instead of the more recent data, which is ok in some cases, but not so great in others.

@k88hudson We could add && !broadcast if that's the only issue. What about removing refresh entirely from new tab load and defer to SYSTEM_TICK?

piatra · 2017-10-13T20:10:39Z

I just realized Highlights probably shouldn't refresh on any actions that TopSitesFeed is also watching; otherwise, we end up with the double updates as we do now when TOP_SITES_UPDATED comes in.

That's what I was trying to do with this check, before calling fetchHiglights on TOP_SITES_UPDATED

on TOP_SITES_UPDATED, HighlightsFeed should only call fetchHighlights if we know some items will get deduped (removed).
The only time we need to bypass that is when Highlights.rows.length === 0 because we returned early when Topsites weren't initialized.

There is the case where you unpin something and it falls out of your Topsites (which shouldn't even happen #3676) case in which maybe a Highlight should show up. But having that show up at the next refresh seems totally resonable (to simplify the logic).

Mardak · 2017-10-13T20:40:00Z

One drawback of highlightsWillDedupe approach is that it conditionally fetches twice. In a state where that method returns true, e.g., a top site was bookmarked (now a bug with the implementation in the PR because bookmarks don't dedupe), every TOP_SITES_UPDATED will do a 2nd fetch for the next 5 days. So in the best case for a given action, there's one fetch and worst case there's two.

So a different approach that could make it in the worst case 1 fetch would perhaps be better.

piatra · 2017-10-13T20:58:49Z

Instead of checking TopSites.rows and Highlights.rows for matching URLs I could directly call dedupe and see if the length of the result is different.
It should definitely be better than invalidating the cache & doing a new query.

Mardak · 2017-10-13T23:54:10Z

One thing I just noticed from my own browsing is that because we currently broadcast fetchHighlights on a NEW_TAB_LOAD, often times I end up seeing highlights without images because it only just realized there are new highlights and should fetch an image. But by doing it some time other than TAB_LOAD, there's a higher chance that the image will already be fetched.

Mardak · 2017-10-14T00:01:30Z

system-addon/lib/HighlightsFeed.jsm

-          this.fetchHighlights(true);
-        } else if (Date.now() - this.highlightsLastUpdated >= HIGHLIGHTS_UPDATE_TIME) {
+      case at.SYSTEM_TICK:
+        if (Date.now() - this.highlightsLastUpdated >= HIGHLIGHTS_UPDATE_TIME) {


There's 3 different "timers" that are being considered here (and for top sites):

system tick interval (5 minutes)

feed last updated (15 minutes)

links cache expiration (5 minutes)

Does it make sense to still have all 3? Or maybe a related question is what's each of their purpose now?

Another issue with similar but slightly different timers is that if they're a little bit off, an expected 5 minute refresh gets skipped and is only actually refreshed after 10 minutes. E.g., due to async stuff, last update was at T+5.01 and the next system tick at T+10.0 is only 4.99 minutes after the last update.

I think that HIGHLIGHTS_UPDATE_TIME is the only one not needed anymore. Looks like a work around to avoid frequent updates on every page load.

system tick interval is needed because this is the new event we react to, independent of page loads. This should also fix the problem of refreshing highlights before having the image ready as described in comment

links cache expiration ensures we get new highlights regardless of special events (like BOOKMARKS_REMOVED/ADDED) that invalidate the cache or our different invalidation tactics (every LAST_FEED_UPDATED minutes).

I'm not sure the timing issue is very important (maybe I don't get it) but we force update on special events for blocking and bookmarking where we need to ensure consistent information. If we don't update the highlights with visits once in a while it should be ok.

The "don't update the highlights with visits once in a while" is actually "every other time" if the timing is just off. It's just being clear that the code might say "refresh every 5 minutes" but turns out to be "refresh every 10 minutes." That may be acceptable, but if 5 minutes was picked for a particular reason, it shouldn't just accidentally be twice as long.

Could we move

this.highlightsLastUpdated = Date.now();

At the top of the fetchHighlights method? I've tried it and it seems to consistently trigger updates at the right moment.
Or maybe change HIGHLIGHTS_UPDATE_TIME to be 3 * SYSTEM_TICK. Instead of counting on timestamp difference we could count modulo 3 and trigger refresh.

I thought you were suggesting HIGHLIGHTS_UPDATE_TIME wouldn't be necessary, so there shouldn't be a need to track highlightsLastUpdated ? We could probably just set LinksCache update time to be 10% less than SYSTEM_TICK, i.e., 4.5 minutes, if we're sure we just want to update at least once every 5 minutes.

Mardak · 2017-10-14T00:13:16Z

system-addon/lib/ActivityStreamMessageChannel.jsm


+  dispatchInitLoadEvent() {


Nice that this separated so easily. I would name this something else as there's already too many "dispatch" and "init"s. Probably something along the lines of simulateNewTabEvents or simulateMessagesForExistingTabs ?

Mardak · 2017-10-14T00:19:00Z

system-addon/lib/HighlightsFeed.jsm

+    const highlightsIndex = SectionsManager.sections.get(SECTION_ID).order;
+    const rows = this.store.getState().Sections[highlightsIndex].rows;
+    // No data available, we need to fetch and dedupe results.
+    if (!rows.length) {


This special case seems to do the exact opposite of what the method says it should be doing. If there's no highlights, there's actually nothing to be deduped, i.e., highlights will not be deduped.

Problem is I kept adding logic to it as new edge cases emerged. Logic should be fine, maybe rename it to something like shouldFetchHighlights similar to shouldUpdate in React when it receives new props.

Mardak · 2017-10-14T00:26:10Z

system-addon/lib/TopSitesFeed.jsm

-          this.refresh(action.meta.fromTarget);
+      case at.SYSTEM_TICK:
+        if (Date.now() - this.lastUpdated >= UPDATE_TIME) {
+          this.refresh();


I believe @k88hudson has mentioned that we want to avoid "random" updates to already opened pages. Looks like we probably want some way to refresh without dispatching to content now and only update chrome… maybe just SendToMain?

@Mardak This applies to Highlights as well, right?

Mardak · 2017-10-14T00:28:22Z

system-addon/test/unit/lib/ActivityStreamMessageChannel.test.js

-          url: "about:sheep",
-          loaded: true,
-          portID: "loaded"
+      describe("#dispatchInitLoadEvent", () => {


This should be at the same level as #createChannel and would avoid indenting everything.

Mardak

Let's remove the feed update time stuff and shorten LinksCache to be less than SYSTEM_TICK. Also would be good to clean up the bool arg APIs without defaults.

With those, this should be good to land

Mardak · 2017-10-17T08:52:44Z

system-addon/lib/HighlightsFeed.jsm

@@ -55,23 +55,18 @@ this.HighlightsFeed = class HighlightsFeed {
  }

  async fetchHighlights(broadcast = false) {
+    // We need TopSites to have been initialised for deduping.
+    // We can return early because we will get another chance to initialize
+    // on TOP_SITES_UPDATED.


nit: odd wrapping and probably can just be shorter while still clear with the code below it:
// We need TopSites for deduping, so just wait for TOP_SITES_UPDATED

Mardak · 2017-10-17T08:52:44Z

system-addon/lib/HighlightsFeed.jsm

          // If the last time we refreshed the data is greater than 15 minutes, fetch again.
-          this.fetchHighlights(false);
+          this.fetchHighlights(true);


We don't want to broadcast on SYSTEM_TICK and don't want feed update time logic.

Mardak · 2017-10-17T08:52:44Z

system-addon/lib/HighlightsFeed.jsm

@@ -162,7 +154,8 @@ this.HighlightsFeed = class HighlightsFeed {
        this.fetchHighlights(false);
        break;
      case at.TOP_SITES_UPDATED:
-        this.fetchHighlights(false);
+        // Only broadcast on the first TOP_SITES_UPDATED call.
+        this.fetchHighlights(false || this.highlightsLastUpdated === 0);


The false || is useless with the other condition. But looking at this closer, wouldn't we want to broadcast the first fetch independent of which action triggered as opposed to only for TOP_SITES_UPDATED?

How about checking this.store.getState().Sections[Highlights].initialized here and dispatch when false.
Maybe wrap everything in a if highlights.length > 0 as highlights can be empty for fresh profiles.

That seems reasonable to switch to broadcasting if not previously initialized. Also noticed that highlightsLength should be unused now that we don't force fetch if fewer than 9 highlights, so that can be removed.

Not entirely sure what you're suggesting for checking length > 0? Don't call updateSection if there's nothing?

Yes, don't broadcast if there's nothing. But then it will not initialize & never show the empty state as that is initialized && !rows.length so nvm.

Mardak · 2017-10-17T08:52:44Z

system-addon/lib/Store.jsm

@@ -138,6 +138,8 @@ this.Store = class Store {
    if (initAction) {
      this.dispatch(initAction);
    }
+
+    this._messageChannel.simulateMessagesForExistingTabs();


A comment for why this is here relative to the other code

Mardak · 2017-10-17T08:52:44Z

system-addon/lib/TopSitesFeed.jsm

   */
-  async refresh(target = null) {
+  async refresh(broadcast = true) {


It's a bit confusing that this is default true, but HighlightsFeed has it default false although it looks like all fetchHighlights calls are explicitly passing in a value instead of relying on defaults. Maybe we should just remove the defaults from both places.

I suppose if we're touching the callers, might as well clean up the API to not pass in a plain boolean arg, so something more like:
refresh(false) -> refresh({broadcast: false})
("refresh... not?" vs "refresh without broadcast")

Mardak · 2017-10-17T08:52:44Z

system-addon/lib/TopSitesFeed.jsm

-          this.refresh(action.meta.fromTarget);
+      case at.SYSTEM_TICK:
+        if (Date.now() - this.lastUpdated >= UPDATE_TIME) {
+          // No update target, no broadcast.


Like Highlights, remove the update time stuff. Also, comment is outdated and probably won't be necessary if we have refresh({broadcast: false})

Mardak · 2017-10-17T08:59:05Z

One thing I noticed when testing the PR, but this probably should be its own issue/PR. #3675 added expiring the highlights links cache on BOOKMARK_ADDED. A new profile causes adds of several bookmarks causing the cache to be expired and fetched from places multiple times on startup.

Mardak

whew! Thanks!

Mardak · 2017-10-17T22:52:31Z

system-addon/lib/HighlightsFeed.jsm

-    // We broadcast when we want to force an update, so get fresh links
-    if (broadcast) {
-      this.linksCache.expire();
+  async fetchHighlights(options = {}) {


Let's describe the option we expect and for TopSites too.

Mardak · 2017-10-17T22:52:31Z

system-addon/lib/HighlightsFeed.jsm

-    this.highlightsLastUpdated = Date.now();
-    this.highlightsLength = highlights.length;
+    const sectionIndex = SectionsManager.sections.get(SECTION_ID).order;
+    const initialized = this.store.getState().Sections[sectionIndex].initialized;


nit: const {initialized} = …

Mardak · 2017-10-17T22:52:31Z

system-addon/lib/HighlightsFeed.jsm

-    this.highlightsLength = highlights.length;
+    const sectionIndex = SectionsManager.sections.get(SECTION_ID).order;
+    const initialized = this.store.getState().Sections[sectionIndex].initialized;
+    // Broadcast when required or if it is the first fetch.


nit: "first fetch" -> "first update" as INIT is most likely the first fetch.

Mardak · 2017-10-17T22:52:31Z

system-addon/lib/LinksCache.jsm

@@ -5,7 +5,10 @@

 this.EXPORTED_SYMBOLS = ["LinksCache"];

-const EXPIRATION_TIME = 5 * 60 * 1000; // 5 minutes
+// This should be slightly less than SYSTEM_TICK_INTERVAL as timers are
+// not precise enough causing skips in the refresh interval.


nit: It's because timer comparisons are too exact ;) and the underlying issue is that async/await functionality will make the last recorded times a little bit later. Might be useful to include the 10%, e.g., "slightly less (10%)"

Mardak · 2017-10-17T22:52:31Z

system-addon/test/unit/lib/HighlightsFeed.test.js

+
+      assert.calledOnce(feed.linksCache.expire);
+    });
+    it("should not call fetchHighlights with broadcast on initialization", () => {


nit: this doesn't seem to describe the test

Mardak · 2017-10-17T22:52:31Z

system-addon/test/unit/lib/TopSitesFeed.test.js

-      sinon.stub(feed, "refresh");
-      feed.onAction(newTabAction);
-      assert.notCalled(feed.refresh);
+    it("should call refresh on SYSTEM_TICK with correct params", () => {


This seems to do the exact same thing as the previous test.

as-pine-proxy · 2017-10-18T15:06:14Z

Treeherder: dfab603038f706e8f8945d1a3c9c8b0a9fc2f6a8

piatra added the PR / Needs review label Oct 12, 2017

piatra assigned k88hudson Oct 12, 2017

piatra requested a review from k88hudson Oct 12, 2017

piatra requested review from Mardak and removed request for k88hudson Oct 12, 2017

piatra assigned piatra and unassigned k88hudson Oct 12, 2017

piatra removed the request for review from Mardak Oct 12, 2017

piatra added PR / In progress don't review yet and removed PR / Needs review labels Oct 12, 2017

Mardak mentioned this pull request Oct 12, 2017

Unexpected behaviors when actions happen before INIT #3697

Closed

k88hudson reviewed Oct 13, 2017

View changes

piatra force-pushed the piatra:bug-3638-reduce-calls branch from f7ca209 to 91b20ea Oct 13, 2017

Mardak reviewed Oct 14, 2017

View changes

Mardak suggested changes Oct 17, 2017

View changes

piatra mentioned this pull request Oct 17, 2017

Multiple calls to linksCache.expire/fetchHighlights on startup in a new profile #3724

Closed

piatra added 10 commits Oct 11, 2017

Reduce number of fetches on startup

3dff3f0

Fix tests

751a91c

Only broadcast the first TOP_SITES_UPDATED

7c4147d

Refactor INIT/LOAD msg calls

ddf788c

TopSitesFeed doesn't broadcast on SYSTEM_TICK

191d2cc

Update comments and method name

94bd2a9

Remove target parameter from TopSitesFeed refresh

d90484a

No need to check TopSitesFeed if we don't broadcast

020ead4

Update tests

60e8261

Remove lastUpdated logic and update tests.

cd1cda2

piatra force-pushed the piatra:bug-3638-reduce-calls branch from 1131a17 to cd1cda2 Oct 17, 2017

typo

77a4982

Mardak approved these changes Oct 17, 2017

View changes

Mardak added PR / R+ with nits and removed PR / In progress don't review yet labels Oct 17, 2017

Update with review comments

6080a17

piatra merged commit 8850b8b into mozilla:master Oct 18, 2017
1 check passed

1 check passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details

piatra deleted the piatra:bug-3638-reduce-calls branch Oct 18, 2017

This was referenced Oct 18, 2017

New profiles trigger a bunch of places queries for highlights #3730

Closed

Be smarter about when to expire thumbnails (especially for Highlights) #3646

Closed

as-pine-proxy added the pushed-to-pine label Oct 18, 2017

piatra mentioned this pull request Oct 24, 2017

[57] The Firefox firstrun page is set as a highlight when first opening the AS New Tab Page #3728

Closed

Mardak mentioned this pull request Oct 25, 2017

Failed thumbnails get requested every 5 minutes #3763

Closed

Mardak mentioned this pull request Nov 10, 2017

Permaorange Ccov tp5o_webext | application crashed [@ libc-2.23.so + 0x35428] #3826

Closed

Mardak mentioned this pull request Apr 11, 2018

Fix Bug 1450875 - Wait for Sections to be added before broadcasting content #4086

Merged

mozilla / activity-stream Archived

Reduce duplicates places queries #3695

Reduce duplicates places queries #3695

Conversation

piatra commented Oct 12, 2017 • edited by Mardak

Mardak commented Oct 12, 2017

piatra commented Oct 12, 2017

piatra commented Oct 12, 2017 • edited

Mardak commented Oct 12, 2017

Mardak commented Oct 12, 2017

Mardak commented Oct 12, 2017

piatra commented Oct 12, 2017

Mardak commented Oct 12, 2017

This comment has been minimized.

k88hudson Oct 13, 2017 • edited Member

This comment has been minimized.

piatra Oct 13, 2017 Author Collaborator

piatra commented Oct 13, 2017

Mardak commented Oct 13, 2017

piatra commented Oct 13, 2017

Mardak commented Oct 13, 2017

This comment has been minimized.

Mardak Oct 14, 2017 Member

This comment has been minimized.

piatra Oct 16, 2017 Author Collaborator

This comment has been minimized.

Mardak Oct 16, 2017 Member

This comment has been minimized.

piatra Oct 17, 2017 • edited Author Collaborator

This comment has been minimized.

Mardak Oct 17, 2017 Member

This comment has been minimized.

Mardak Oct 14, 2017 • edited Member

This comment has been minimized.

Mardak Oct 14, 2017 Member

This comment has been minimized.

piatra Oct 16, 2017 Author Collaborator

This comment has been minimized.

Mardak Oct 14, 2017 Member

This comment has been minimized.

piatra Oct 16, 2017 • edited Author Collaborator

This comment has been minimized.

Mardak Oct 14, 2017 Member

Mardak left a comment

This comment has been minimized.

Mardak Oct 17, 2017 Member

This comment has been minimized.

Mardak Oct 17, 2017 Member

This comment has been minimized.

Mardak Oct 17, 2017 Member

This comment has been minimized.

piatra Oct 17, 2017 Author Collaborator

This comment has been minimized.

Mardak Oct 17, 2017 Member

This comment has been minimized.

piatra Oct 17, 2017 • edited Author Collaborator

This comment has been minimized.

Mardak Oct 17, 2017 Member

This comment has been minimized.

Mardak Oct 17, 2017 Member

This comment has been minimized.

Mardak Oct 17, 2017 Member

This comment has been minimized.

Mardak Oct 17, 2017 Member

Mardak commented Oct 17, 2017

Mardak left a comment

This comment has been minimized.

Mardak Oct 17, 2017 Member

This comment has been minimized.

Mardak Oct 17, 2017 Member

This comment has been minimized.

Mardak Oct 17, 2017 Member

This comment has been minimized.

Mardak Oct 17, 2017 Member

This comment has been minimized.

Mardak Oct 17, 2017 Member

This comment has been minimized.

Mardak Oct 17, 2017 Member

as-pine-proxy commented Oct 18, 2017

piatra commented Oct 12, 2017 •

edited by Mardak

piatra commented Oct 12, 2017 •

edited

k88hudson Oct 13, 2017 •

edited

Member

piatra Oct 13, 2017
Author Collaborator

Mardak Oct 14, 2017
Member

piatra Oct 16, 2017
Author Collaborator

Mardak Oct 16, 2017
Member

piatra Oct 17, 2017 •

edited

Author Collaborator

Mardak Oct 17, 2017
Member

Mardak Oct 14, 2017 •

edited

Member

Mardak Oct 14, 2017
Member

piatra Oct 16, 2017
Author Collaborator

Mardak Oct 14, 2017
Member

piatra Oct 16, 2017 •

edited

Author Collaborator

Mardak Oct 14, 2017
Member

Mardak Oct 17, 2017
Member

Mardak Oct 17, 2017
Member

Mardak Oct 17, 2017
Member

piatra Oct 17, 2017
Author Collaborator

Mardak Oct 17, 2017
Member

piatra Oct 17, 2017 •

edited

Author Collaborator

Mardak Oct 17, 2017
Member

Mardak Oct 17, 2017
Member

Mardak Oct 17, 2017
Member

Mardak Oct 17, 2017
Member

Mardak Oct 17, 2017
Member

Mardak Oct 17, 2017
Member

Mardak Oct 17, 2017
Member

Mardak Oct 17, 2017
Member

Mardak Oct 17, 2017
Member

Mardak Oct 17, 2017
Member