One Zero bank test failed

[jond01]

https://github.com/jond01/xil/actions/runs/6708383613

After a quick examination - there are probably two separate issues.

Need to understand if the following assertions are correct:

xil/tests/test_onezero.py

Lines 19 to 24 in 4f21d82

	assert (
	df[("transfer", "buy")] < df[("currency", "official rate")]
	).all(), "The buy rate is not lower than the official rate"
	assert (
	df[("currency", "official rate")] < df[("transfer", "sell")]
	).all(), "The sell rate is not higher than the official rate"

The "official rate" is probably updated once a day, so the sell/buy rates may not respect the assumed relations at all times.

The raw JSON from a simple scraping is different than the one I see on the web browser.
The scraped one, which is used in XIL, can be obtained with:

import urllib.request

with urllib.request.urlopen("https://dv16ymfyh91nr.cloudfront.net/MarketingRatesReport/MarketingSiteFCYRatesCurrentReport.json") as f:
    print(f.read().decode())

The difference is:

{
    "generatingReportDateTime": "2023-10-31",
    "marketingRecords": [
        {
             "fromCurrency": "EUR",
             "toCurrency": "ILS",
             "representativeExchangeRate": 0.2329,
-            "buyRate": 0.2326,
-            "sellRate": 0.2349
+            "buyRate": 0.2318,
+            "sellRate": 0.2341
         },
         {
             "fromCurrency": "ILS",
             "toCurrency": "EUR",
             "representativeExchangeRate": 4.2934,
-            "buyRate": 4.2573,
-            "sellRate": 4.3001
+            "buyRate": 4.2711,
+            "sellRate": 4.3141
         },
         {
             "fromCurrency": "USD",
             "toCurrency": "ILS",
             "representativeExchangeRate": 0.2466,
-            "buyRate": 0.2467,
-            "sellRate": 0.2492
+            "buyRate": 0.2457,
+            "sellRate": 0.2482
         },
         {
             "fromCurrency": "ILS",
             "toCurrency": "USD",
             "representativeExchangeRate": 4.0550,
-            "buyRate": 4.0135,
-            "sellRate": 4.0538
+            "buyRate": 4.0288,
+            "sellRate": 4.0693
         }
     ]
 }

Need to understand where does this difference data come from.

[jond01]

Re-checking now: there's no difference between the scraped and online-viewed JSON, and the test passed.

[jond01]

After checking manually and reviewing the test runs (including the latest run), the issue does not reproduce.
I'm going to close it as a temporary bug.

[jond01]

https://github.com/jond01/xil/actions/runs/6865423605/job/18674728221

[jond01]

https://github.com/jond01/xil/actions/runs/7330517632
https://github.com/jond01/xil/actions/runs/7339447462
https://github.com/jond01/xil/actions/runs/7378131143
https://github.com/jond01/xil/actions/runs/7432286995

[jond01]

The latest run (from today) shows the following difference when accessing through Python and a web browser (Chrome/Firefox/Safari):

{
-    "generatingReportDateTime": "2024-01-05",
+    "generatingReportDateTime": "2024-01-06",
    "marketingRecords": [
        {
            "buyRate": 0.2475,
            "fromCurrency": "EUR",
            "representativeExchangeRate": 0.2505,
            "sellRate": 0.2499,
            "toCurrency": "ILS"
        },
        {
            "buyRate": 4.001,
            "fromCurrency": "ILS",
            "representativeExchangeRate": 3.9917,
            "sellRate": 4.0412,
            "toCurrency": "EUR"
        },
        {
            "buyRate": 0.2698,
            "fromCurrency": "USD",
            "representativeExchangeRate": 0.2735,
            "sellRate": 0.2725,
            "toCurrency": "ILS"
        },
        {
            "buyRate": 3.6691,
            "fromCurrency": "ILS",
            "representativeExchangeRate": 3.656,
            "sellRate": 3.706,
            "toCurrency": "USD"
        }
    ]
}

The later and true date (2023-01-06) is from Python.
A quick test showed that when passing the following "Accept-Encoding" header in Python, the result was the same as in the web:

import urllib.request

headers = {"Accept-Encoding": "gzip, deflate, br"}  # This header makes the server return an outdated response
url = "https://dv16ymfyh91nr.cloudfront.net/MarketingRatesReport/MarketingSiteFCYRatesCurrentReport.json"

req = urllib.request.Request(url, headers=headers)
with urllib.request.urlopen(req) as f:
    print(f.read().decode())

Original check code

import json
import urllib.request
import logging

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger(__name__)

all_headers = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "en-GB,en-US;q=0.9,en;q=0.8",
    "Connection": "keep-alive",
    "Host": "dv16ymfyh91nr.cloudfront.net",
    "Priority": "u=0, i",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2.1 Safari/605.1.15",
}
url = "https://dv16ymfyh91nr.cloudfront.net/MarketingRatesReport/MarketingSiteFCYRatesCurrentReport.json"

for item in all_headers.items():
    headers = dict([item])
    logger.debug(headers)
    req = urllib.request.Request(url, headers=headers)
    with urllib.request.urlopen(req) as f:
        data = f.read().decode()
    if json.loads(data)["generatingReportDateTime"] == "2024-01-05":
        logger.info("Found it!")

It explains the discrepancy mentioned in the issue's description, but not the failure of the test.

Moreover, since writing this comment, the web browser's response has been updated, matching now Python.

[jond01]

https://github.com/jond01/xil/actions/runs/7432286995/job/20225973783

The client sub-issue (web browser/Python) is orthogonal to the test failure.
Even when the same, the test may fail, as in the case linked above.
The "buyRate" is higher than the "representativeExchangeRate" for both EUR and USD ("toCurrency").
The fix for the issue is probably to remove the wrong assertion.

[jond01]

https://github.com/jond01/xil/actions/runs/8422210010/job/23060978005#step:6:42