Run a smoke test of the new pipeline to ensure 10x capacity throughput

[rviscomi]

Blocked by #2, #52

At the current crawl size, the pipeline should complete in less than 60 hours.

[pmeenan]

Looks like there are some pub/sub bottlenecks to work out.

We have the VM's autoscaling based on test queue length and are currently configured for ~4x the last crawl (with all of the agents running) but the actual test thruput is only 12k/hr across both platforms. Even though we added desktop lighthouse, that's a fraction of what it should be doing (probably 10x that or more).

When it was scaling by CPU utilization it wasn't scaling to anywhere near capacity so it doesn't look like the workers have "enough work" even though there are 12M tests in the pubsub queue.

Going to fire up an agent manually where I can SSH into it and see what is going on. My guess is we're exceeding a pubsub API limit of some kind since we're polling the API from all of the agents but it's not really that high of a rate so it would be surprising.

[pmeenan]

I have .... strong words for pub/sub's usefulness as a "pull" work queue. It looks like it is probably very-much optimized for already having listners and then routing messages dynamically and queuing them for existing listeners (best guess).

I have a couple of dev agents (on in GCE, on in my local VM) polling the crawl-queue queue for messages with a 60-second timeout and they are getting 0 messages from the subscription, even though there are over 12M un-ack'd messages in the queue.

If I can't get it figured out soon I may switch to using cloud Redis instead which has message queues that actually work (there's more config envolved though)

[pmeenan]

Been an interesting day but I think we're in better shape now. Currently running 8400 VM's which is in the neighborhood of 12x where we started.

pub/sub thruput seems to be coming and going but I'll stop throwing new instances at it for a while to see if it stabilizes. Even with lighthouse enabled for desktop we peaked a little over 250k tests per hour. Now just have to see if pub/sub can sustain it or if even async falls over with that many instances.

[pmeenan]

12 hours into the tweaked config and things are looking solid. Running around 3.76M URLs per day with the retry queue working as expected. It will take a bit more than 60 hours since the crawl start because if the issues during the first 12 hours or so and because desktop now includes lighthouse but I'd call it a successful test (well, really, running the production crawl at full 10x speed, not just a test).

The HAR GCS pipeline pub/sub queue is also consistently keeping up with the flow of messages (which hopefully means they are being processed as well).

Will wait until the crawl is complete before closing just to make sure that everything flushes through cleanly.

[rviscomi]

We started running into TypeError: Integer exceeds 64-bit range errors Tuesday evening. I spoke with @giancarloaf who thinks this may have to do with the hashed requestid and pageid fields. These were only created for parity with the MySQL tables and I'm of the opinion that we should stop generating them, which may help to avoid this bug. It's possible that other fields may be causing the error.

The pageid field is still necessary for joining the summary_pages and summary_requests field. To work around that, I'd suggest that we add a page field to both tables representing the initial URL of the page being tested. This field should be joinable with the origins in CrUX and not represent the final document URL after any redirects. Eventually I'd like us to standardize on "page" referring to the URL of the page, and "url" referring to the URL of the requests, similar to how it's currently used in the almanac dataset. This is a breaking change though for anyone with queries that join on pageid, so we should be sure we want to go in this direction. Any thoughts @tunetheweb or @paulcalvano?

[pmeenan]

Sounds like something to do at processing time, is there something you need me to do? The metadata has an integer page_id which is just an increasing index that is incremented as URLs are submitted and isn't guaranteed to be consistent from one crawl to the next but it is guaranteed to be unique within a given crawl.

For request ID, we could easily just bitfield the 64-bit integer space and use the high 32-bits for the page ID and the low 32 bits for the request number within a given page. That way we can have 4B pages, each with 4B requests and still guarantee they will be unique.

[rviscomi]

Oooooh I didn't realize that was added to the metadata. That would work. @giancarloaf WDYT?

Request IDs are entirely useless since there's nothing to join them with, so returning null wouldn't break anything.

Edit: I should add that they might be useful to distinguish between repeated requests on the same page. But I think something like a simple array index would be more useful to be able to tell which came first.

[tunetheweb]

Request IDs are entirely useless since there's nothing to join them with, so returning null wouldn't break anything.

Edit: I should add that they might be useful to distinguish between repeated requests on the same page. But I think something like a simple array index would be more useful to be able to tell which came first.

Was about to say. We should always have a unique id even if it’s a combination of fields (which pageid + url is not). Happy if we have an array index for all requests within a page.

[giancarloaf]

Sounds like something to do at processing time, is there something you need me to do? The metadata has an integer page_id which is just an increasing index that is incremented as URLs are submitted and isn't guaranteed to be consistent from one crawl to the next but it is guaranteed to be unique within a given crawl.

Excellent! If it's already available, I would prefer to use that for consistency. Can you point me to the json path for page_id in the metadata? Is that currently available in GCS outputs available for May?

---

For request ID, we could easily just bitfield the 64-bit integer space and use the high 32-bits for the page ID and the low 32 bits for the request number within a given page. That way we can have 4B pages, each with 4B requests and still guarantee they will be unique.

Request IDs are entirely useless since there's nothing to join them with, so returning null wouldn't break anything.
Edit: I should add that they might be useful to distinguish between repeated requests on the same page. But I think something like a simple array index would be more useful to be able to tell which came first.

Was about to say. We should always have a unique id even if it’s a combination of fields (which pageid + url is not). Happy if we have an array index for all requests within a page.

requestid is currently left null in the new pipeline. I have no preference, but @pmeenan's suggestion should be possible, something like:

page_id = 123
request_num = 456
(page_id << 32) + request_num
# 528280977864

[pmeenan]

The page_id is right next to the rank and layout fields in the page-level metadata (and has been there for ~2 crawls):

                "_metadata": {
                    "rank": 10000000,
                    "page_id": 1491685,
                    "layout": "Desktop",
                    "crawl_depth": 0,
                    "link_depth": 0
                },

FWIW, that is also where the primary/secondary pages fields are.

crawl_depth - 0 for the first pass, 1 for child pages of the main page, 2 for children of the children (if we ever go deeper)
link_depth - 0 for the first child page, 1 for the second child page, etc.

The main page will always have 0/0 and all child pages will have a non-zero crawl_depth.

On the request id's, each request in the HAR has 2 sequence numbers:

                "_index": 1,
                "_number": 2

They are both the same, _index is just 0-indexed and _number starts at 1.

The Request ID's could be what @giancarloaf mentioned above, combining page_id from the page it is on and _number from the request field. which would give us a globally unique ID (within the crawl) for the request record.

[rviscomi]

The streaming summary pipeline finished successfully this month so I think we can close this out!