Use single https session to retrieve all rock-on definitions #2707
Comments
|
I'll have a little experiment with this as having first considered server side compression, I think we would be far better off looking to http session refinements, give he actual download is one of the smallest time sinks. Plus we would inevitably incur decompression time on the client: especially on lower power devices. |
|
Notes re timings: With single session:
Removing proposed session manager:
From the above experimental change, which just wraps our Rock-on definition files retrieval in a requests.Session() manager, we look to have a significant speed-up, taking a little less than half the time to achieve the same number of gets. Looking into better logging of just the retrieval to see the real difference. |
|
More focused timing experiment but still including the initial parsing to a single dictionary: Before proposalI.e. https session for each json:
After proposalTo retrieve all definitions under the same https session:
We take: (6.33/19.23) = 0.33 around a third of the time. |
…#2707 By sharing a single https session, instantiated on the first GET, we avoid major cumulative delays and reduce client/server resource overhead. Note that we still employ a dedicated https session to retrieve our 'index' file of 'root.json'. ## Includes - Minor fstring adoption in modified area of code. - Info log of definitions retrieval and initial parsing time. Intended as a diagnostic aid, and to assist with field feedback on this process of indeterminate duration. - Individual file retrieval timeout reduced to 5 seconds.
…-to-retrieve-all-rock-on-definitions Use single https session to retrieve all rock-on definitions #2707
|
Closing as: |
Whenever we populate/update available rock-ons we first retrieve root.json, then parse this file and create a new https session to retrieve each individual json Rock-on definition file listed within root.json. The vast majority of the time taken in retrieving each Rock-on definition is spent in the session setup itself. I.e. with our 70+ Rock-ons we need to create 71+ https sessions.
It is proposed that we consider instead maintaining our initial root.json retrieval session, useful to establish if a connection is even possible, then attempt to retrieve all actual definition files over a follow-up session. Where our expected time saving will be 70+ fold on individual sessions.
I.e. To retrieve a large Rock-on definition (3.9KB) via a Web-browser is as follows:
'''
Started at 6.60 ms
Resource Scheduling DURATION
Queueing
6.60 ms
Connection Start DURATION
Stalled
0.51 ms
DNS Lookup
16 μs
Initial connection
159.28 ms
SSL
74.30 ms
Request/Response DURATION
Request sent
0.13 ms
Waiting for server response
72.87 ms
Content Download
0.91 ms
Explanation 240.41 ms
The text was updated successfully, but these errors were encountered: