Issue loading data from FRED #157
Comments
|
Hey Howard! Thanks, it looks like it's an SSL error on the FRED side. This is the error in your traceback: Or this That's the first time I see it, but I did see an error with FRED on one of the GH test runners yesterday, it worked fine after re-running. FYI the endpoint is something like this |
|
FYI @hhenson, the example you're running only uses FRED to get the USD risk free (central bank) rate, Yahoo Finance is working fine for you it seems. You can get around that by disabling that part of the the market data download. Substitute this line with this that sets the risk free rate to zero... |
|
Also, this thread https://stackoverflow.com/questions/52805115/certificate-verify-failed-unable-to-get-local-issuer-certificate seems to suggest you have to run some post-installation script to enable SSL for your local Python, it looks like you're using one in the system libraries. I normally suggest using brew for installing Python on mac... |
see GH issue #157 now it goes through requests instead of using pd.read_csv(endpoint) directly; should have no effect on performance but it seems requests is more robust in handling ssl (should use certifi)
|
OK, was using a pyenv setup, changed to use the brew system installation and got it working! |
|
FYI FIRST RUN ################################################################# Avg. return (annualized) 60.8% Avg. growth rate (annualized) 29.7% Avg. StocksTransactionCost 0bp Sharpe ratio 0.75 Avg. drawdown -48.1% Avg. policy time 0.152s SECOND RUN ################################################################# Avg. return (annualized) 60.8% Avg. growth rate (annualized) 29.6% Avg. StocksTransactionCost 0bp Sharpe ratio 0.75 Avg. drawdown -48.1% Avg. policy time 0.107s This is on an M3 MacBook Pro with the Max chipset using the brew version of Python 3.11.9 |
|
FIRST RUN ################################################################# Avg. return (annualized) 60.8% Avg. growth rate (annualized) 29.7% Avg. StocksTransactionCost 0bp Sharpe ratio 0.75 Avg. drawdown -48.1% Avg. policy time 0.198s SECOND RUN ################################################################# Avg. return (annualized) 60.8% Avg. growth rate (annualized) 29.7% Avg. StocksTransactionCost 0bp Sharpe ratio 0.75 Avg. drawdown -48.1% Avg. policy time 0.125s This is on the M2 ULTRA Studio |
|
No Issue |
|
Very nice, second runs are the most interesting ones. That example uses an open-source solver (ECOS), which is very similar to MOSEK. Some (~10% I guess) improvement can come from optimizing the "result time", which could be done better (but it's very tricky to make sure the accounting is correct). Otherwise it is IMO about as fast as it can be done. *edit: ECOS, not CLARABEL |
|
@hhenson I just re-ran it myself, changing the solver to the new open-source one that is default in CVXPY 1.5; interesting to note that it gives slightly lower Sharpe, but it is a bit faster. |
Trying to run the performance example, it loads the SNP500 data, but it fails when trying to load the FRED data.
This is the output:
Updating data.......................................................YahooFinance("BBWI") has dubious total open-to-open returns for timestamps: DatetimeIndex(['1982-11-23 14:30:00+00:00', '1985-06-27 13:30:00+00:00'], dtype='datetime64[ns, UTC]', freq=None)
...YahooFinance("BEN") has dubious total open-to-open returns for timestamps: DatetimeIndex(['1984-11-12 14:30:00+00:00', '1984-11-14 14:30:00+00:00',
'1984-11-15 14:30:00+00:00', '1984-11-19 14:30:00+00:00',
'1984-12-28 14:30:00+00:00', '1985-01-02 14:30:00+00:00'],
dtype='datetime64[ns, UTC]', freq=None)
...YahooFinance("BIIB") has dubious total open-to-open returns for timestamps: DatetimeIndex(['2005-02-25 14:30:00+00:00'], dtype='datetime64[ns, UTC]', freq=None)
....................................................................YahooFinance("D") has dubious total open-to-open returns for timestamps: DatetimeIndex(['1983-05-23 13:30:00+00:00'], dtype='datetime64[ns, UTC]', freq=None)
...................................................................YahooFinance("FRT") has dubious total open-to-open returns for timestamps: DatetimeIndex(['1974-05-24 13:30:00+00:00', '1974-05-29 13:30:00+00:00',
'1974-06-04 13:30:00+00:00'],
dtype='datetime64[ns, UTC]', freq=None)
...........YahooFinance("GL") has dubious total open-to-open returns for timestamps: DatetimeIndex(['1984-06-08 13:30:00+00:00', '1985-10-10 13:30:00+00:00'], dtype='datetime64[ns, UTC]', freq=None)
...............YahooFinance("HD") has dubious total open-to-open returns for timestamps: DatetimeIndex(['1983-03-18 14:30:00+00:00'], dtype='datetime64[ns, UTC]', freq=None)
.........................................YahooFinance("JNPR") has dubious total open-to-open returns for timestamps: DatetimeIndex(['2024-01-08 14:30:00+00:00'], dtype='datetime64[ns, UTC]', freq=None)
.............YahooFinance("KR") has dubious total open-to-open returns for timestamps: DatetimeIndex(['2017-06-15 13:30:00+00:00'], dtype='datetime64[ns, UTC]', freq=None)
............YahooFinance("LOW") has dubious total open-to-open returns for timestamps: DatetimeIndex(['1983-04-29 13:30:00+00:00'], dtype='datetime64[ns, UTC]', freq=None)
............YahooFinance("MCD") has dubious total open-to-open returns for timestamps: DatetimeIndex(['1969-07-09 13:30:00+00:00'], dtype='datetime64[ns, UTC]', freq=None)
.............YahooFinance("MMC") has dubious total open-to-open returns for timestamps: DatetimeIndex(['1980-11-21 14:30:00+00:00', '2004-10-14 13:30:00+00:00'], dtype='datetime64[ns, UTC]', freq=None)
.........................................................................YahooFinance("PSA") has dubious total open-to-open returns for timestamps: DatetimeIndex(['1981-01-28 14:30:00+00:00', '1981-02-24 14:30:00+00:00'], dtype='datetime64[ns, UTC]', freq=None)
.............................................YahooFinance("SYY") has dubious total open-to-open returns for timestamps: DatetimeIndex(['1975-01-10 14:30:00+00:00', '1975-01-29 14:30:00+00:00',
'1977-05-11 13:30:00+00:00', '1977-05-12 13:30:00+00:00'],
dtype='datetime64[ns, UTC]', freq=None)
..................YahooFinance("TRV") has dubious total open-to-open returns for timestamps: DatetimeIndex(['1986-06-12 13:30:00+00:00'], dtype='datetime64[ns, UTC]', freq=None)
...YahooFinance("TSN") has dubious total open-to-open returns for timestamps: DatetimeIndex(['1983-04-18 14:30:00+00:00', '1985-04-24 14:30:00+00:00'], dtype='datetime64[ns, UTC]', freq=None)
.........YahooFinance("UHS") has dubious total open-to-open returns for timestamps: DatetimeIndex(['1986-10-21 13:30:00+00:00'], dtype='datetime64[ns, UTC]', freq=None)
.....YahooFinance("URI") has dubious total open-to-open returns for timestamps: DatetimeIndex(['2007-11-14 14:30:00+00:00'], dtype='datetime64[ns, UTC]', freq=None)
............YahooFinance("VTRS") has dubious total open-to-open returns for timestamps: DatetimeIndex(['1983-07-01 13:30:00+00:00', '1985-02-19 14:30:00+00:00'], dtype='datetime64[ns, UTC]', freq=None)
.........................
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 1344, in do_open
h.request(req.get_method(), req.selector, req.data, headers,
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1331, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1377, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1326, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1085, in _send_output
self.send(msg)
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1029, in send
self.connect()
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/http/client.py", line 1472, in connect
self.sock = self._context.wrap_socket(self.sock,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/ssl.py", line 455, in wrap_socket
return self.sslsocket_class._create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/ssl.py", line 1042, in _create
self.do_handshake()
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/ssl.py", line 1320, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/.venv/lib/python3.12/site-packages/cvxportfolio/data/symbol_data.py", line 1090, in _internal_download
return pd.to_numeric(pd.read_csv(
^^^^^^^^^^^^
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/.venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
return _read(filepath_or_buffer, kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/.venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 620, in _read
parser = TextFileReader(filepath_or_buffer, **kwds)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/.venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1620, in init
self._engine = self._make_engine(f, self.engine)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/.venv/lib/python3.12/site-packages/pandas/io/parsers/readers.py", line 1880, in _make_engine
self.handles = get_handle(
^^^^^^^^^^^
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/.venv/lib/python3.12/site-packages/pandas/io/common.py", line 728, in get_handle
ioargs = _get_filepath_or_buffer(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/.venv/lib/python3.12/site-packages/pandas/io/common.py", line 384, in _get_filepath_or_buffer
with urlopen(req_info) as req:
^^^^^^^^^^^^^^^^^
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/.venv/lib/python3.12/site-packages/pandas/io/common.py", line 289, in urlopen
return urllib.request.urlopen(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 215, in urlopen
return opener.open(url, data, timeout)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 515, in open
response = self._open(req, data)
^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 532, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 492, in _call_chain
result = func(*args)
^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 1392, in https_open
return self.do_open(http.client.HTTPSConnection, req,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/urllib/request.py", line 1347, in do_open
raise URLError(err)
urllib.error.URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1000)>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/performance_test.py", line 50, in
simulator = cvx.StockMarketSimulator(UNIVERSE)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/.venv/lib/python3.12/site-packages/cvxportfolio/simulator.py", line 764, in init
super().init(universe=universe,
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/.venv/lib/python3.12/site-packages/cvxportfolio/simulator.py", line 155, in init
self.market_data = DownloadedMarketData(
^^^^^^^^^^^^^^^^^^^^^
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/.venv/lib/python3.12/site-packages/cvxportfolio/data/market_data.py", line 714, in init
self._add_cash_column(self.cash_key, grace_period=grace_period)
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/.venv/lib/python3.12/site-packages/cvxportfolio/data/market_data.py", line 305, in _add_cash_column
data = Fred(
^^^^^
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/.venv/lib/python3.12/site-packages/cvxportfolio/data/symbol_data.py", line 99, in init
self.update(grace_period)
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/.venv/lib/python3.12/site-packages/cvxportfolio/data/symbol_data.py", line 191, in update
updated = self._download(
^^^^^^^^^^^^^^^
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/.venv/lib/python3.12/site-packages/cvxportfolio/data/symbol_data.py", line 1110, in _download
return self._internal_download(symbol)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/hhenson/PycharmProjects/cvxportfolio_experiment/.venv/lib/python3.12/site-packages/cvxportfolio/data/symbol_data.py", line 1094, in _internal_download
raise DataError(f"Download of {symbol}"
cvxportfolio.errors.DataError: Download of DFF from Fred failed. Are you connected to the Internet?
I dug a bit into the code and it seems that the code tries to load data using the URL:
https://fred.stlouisfed.org/graph/fredgraph.csv
If I try and put this into a browser I get:
It seems like it may be a change made to the Fred Website?
May also be geographic issue?
The text was updated successfully, but these errors were encountered: