Potential Speed Increase for Multiple Stocks Download
Summary
I have identified a potential speed increase.
Currently a network operation occurs sequentially, but we could take advantage of the time it takes to fulfill a request to begin another, thus increasing efficiency by having concurrent network requests.
Explanation
scripts/yFinDler_main.js calls function retrieveJson() per ticker entered by user when they do the multiple file download option.
retrieveJson() is a function I created to wrap up the process of connecting to my CORS proxy to download data from yahoo finance.
Despite the fact that I created retrieveJson() to have asynchronous capabilities, I am still currently calling it in a synchronous fashion.
I believe with a slight edit, we could make the requests happen concurrently, thus taking advantage of the waiting time for a request to complete, by sending the next request, rather than waiting for a response from the first request before doing anything.
My main fear is that after this change, using the multiple stock downloading option ends up blasting hundreds of requests very quickly and either overloads my CORS proxy or the user's own browser.
However, after a brief googling, I read that both Firefox and Chrome have a default of 6 concurrent connections to a server, so hopefully that helps.
I also confirmed the Firefox side by checking my own browser's setting in about:config under network.http.max-persistent-connections-per-server, which had a value of 6.
Below is data from a small experiment.
Baseline
This is data from a onetime experiment to get a sense of the current capabilities of the tool.
The list of tickers I used will be placed at the bottom.
As per my tweet, it took approximately 8 minutes and 20 seconds to download the zip file.
The zip file size was approximately 425 megabytes.
I also noted that the browser froze briefly as it was generating the zip file, but the pause wasn't large enough to be worrisome.
Tickers List
A few days before writing this issue, these are the tickers I pulled from Wikipedia,
List of S&P 500 companies.
Two tickers failed because they weren't Yahoo Finance formatted:
Here is the list with no formatting for easy repeating.
MMM
AOS
ABT
ABBV
ACN
ADBE
AMD
AES
AFL
A
APD
ABNB
AKAM
ALB
ARE
ALGN
ALLE
LNT
ALL
GOOGL
GOOG
MO
AMZN
AMCR
AEE
AEP
AXP
AIG
AMT
AWK
AMP
AME
AMGN
APH
ADI
AON
APA
APO
AAPL
AMAT
APP
APTV
ACGL
ADM
ARES
ANET
AJG
AIZ
T
ATO
ADSK
ADP
AZO
AVB
AVY
AXON
BKR
BALL
BAC
BAX
BDX
BRK.B
BBY
TECH
BIIB
BLK
BX
XYZ
BK
BA
BKNG
BSX
BMY
AVGO
BR
BRO
BF.B
BLDR
BG
BXP
CHRW
CDNS
CPT
CPB
COF
CAH
CCL
CARR
CVNA
CASY
CAT
CBOE
CBRE
CDW
COR
CNC
CNP
CF
CRL
SCHW
CHTR
CVX
CMG
CB
CHD
CIEN
CI
CINF
CTAS
CSCO
C
CFG
CLX
CME
CMS
KO
CTSH
COHR
COIN
CL
CMCSA
FIX
CAG
COP
ED
STZ
CEG
COO
CPRT
GLW
CPAY
CTVA
CSGP
COST
CTRA
CRH
CRWD
CCI
CSX
CMI
CVS
DHR
DRI
DDOG
DVA
DECK
DE
DELL
DAL
DVN
DXCM
FANG
DLR
DG
DLTR
D
DPZ
DASH
DOV
DOW
DHI
DTE
DUK
DD
ETN
EBAY
SATS
ECL
EIX
EW
EA
ELV
EME
EMR
ETR
EOG
EPAM
EQT
EFX
EQIX
EQR
ERIE
ESS
EL
EG
EVRG
ES
EXC
EXE
EXPE
EXPD
EXR
XOM
FFIV
FDS
FICO
FAST
FRT
FDX
FIS
FITB
FSLR
FE
FISV
F
FTNT
FTV
FOXA
FOX
BEN
FCX
GRMN
IT
GE
GEHC
GEV
GEN
GNRC
GD
GIS
GM
GPC
GILD
GPN
GL
GDDY
GS
HAL
HIG
HAS
HCA
DOC
HSIC
HSY
HPE
HLT
HD
HON
HRL
HST
HWM
HPQ
HUBB
HUM
HBAN
HII
IBM
IEX
IDXX
ITW
INCY
IR
PODD
INTC
IBKR
ICE
IFF
IP
INTU
ISRG
IVZ
INVH
IQV
IRM
JBHT
JBL
JKHY
J
JNJ
JCI
JPM
KVUE
KDP
KEY
KEYS
KMB
KIM
KMI
KKR
KLAC
KHC
KR
LHX
LH
LRCX
LVS
LDOS
LEN
LII
LLY
LIN
LYV
LMT
L
LOW
LULU
LITE
LYB
MTB
MPC
MAR
MRSH
MLM
MAS
MA
MKC
MCD
MCK
MDT
MRK
META
MET
MTD
MGM
MCHP
MU
MSFT
MAA
MRNA
TAP
MDLZ
MPWR
MNST
MCO
MS
MOS
MSI
MSCI
NDAQ
NTAP
NFLX
NEM
NWSA
NWS
NEE
NKE
NI
NDSN
NSC
NTRS
NOC
NCLH
NRG
NUE
NVDA
NVR
NXPI
ORLY
OXY
ODFL
OMC
ON
OKE
ORCL
OTIS
PCAR
PKG
PLTR
PANW
PSKY
PH
PAYX
PYPL
PNR
PEP
PFE
PCG
PM
PSX
PNW
PNC
POOL
PPG
PPL
PFG
PG
PGR
PLD
PRU
PEG
PTC
PSA
PHM
PWR
QCOM
DGX
Q
RL
RJF
RTX
O
REG
REGN
RF
RSG
RMD
RVTY
HOOD
ROK
ROL
ROP
ROST
RCL
SPGI
CRM
SNDK
SBAC
SLB
STX
SRE
NOW
SHW
SPG
SWKS
SJM
SW
SNA
SOLV
SO
LUV
SWK
SBUX
STT
STLD
STE
SYK
SMCI
SYF
SNPS
SYY
TMUS
TROW
TTWO
TPR
TRGP
TGT
TEL
TDY
TER
TSLA
TXN
TPL
TXT
TMO
TJX
TKO
TTD
TSCO
TT
TDG
TRV
TRMB
TFC
TYL
TSN
USB
UBER
UDR
ULTA
UNP
UAL
UPS
URI
UNH
UHS
VLO
VTR
VLTO
VRSN
VRSK
VZ
VRTX
VRT
VTRS
VICI
V
VST
VMC
WRB
GWW
WAB
WMT
DIS
WBD
WM
WAT
WEC
WFC
WELL
WST
WDC
WY
WSM
WMB
WTW
WDAY
WYNN
XEL
XYL
YUM
ZBRA
ZBH
ZTS
Potential Speed Increase for Multiple Stocks Download
Summary
I have identified a potential speed increase.
Currently a network operation occurs sequentially, but we could take advantage of the time it takes to fulfill a request to begin another, thus increasing efficiency by having concurrent network requests.
Explanation
scripts/yFinDler_main.js calls function retrieveJson() per ticker entered by user when they do the multiple file download option.
retrieveJson() is a function I created to wrap up the process of connecting to my CORS proxy to download data from yahoo finance.
Despite the fact that I created retrieveJson() to have asynchronous capabilities, I am still currently calling it in a synchronous fashion.
I believe with a slight edit, we could make the requests happen concurrently, thus taking advantage of the waiting time for a request to complete, by sending the next request, rather than waiting for a response from the first request before doing anything.
My main fear is that after this change, using the multiple stock downloading option ends up blasting hundreds of requests very quickly and either overloads my CORS proxy or the user's own browser.
However, after a brief googling, I read that both Firefox and Chrome have a default of 6 concurrent connections to a server, so hopefully that helps.
I also confirmed the Firefox side by checking my own browser's setting in
about:configundernetwork.http.max-persistent-connections-per-server, which had a value of 6.Below is data from a small experiment.
Baseline
This is data from a onetime experiment to get a sense of the current capabilities of the tool.
The list of tickers I used will be placed at the bottom.
As per my tweet, it took approximately 8 minutes and 20 seconds to download the zip file.
The zip file size was approximately 425 megabytes.
I also noted that the browser froze briefly as it was generating the zip file, but the pause wasn't large enough to be worrisome.
Tickers List
A few days before writing this issue, these are the tickers I pulled from Wikipedia,
List of S&P 500 companies.
Two tickers failed because they weren't Yahoo Finance formatted:
Here is the list with no formatting for easy repeating.