Incorrect data #30

Closed
nickray opened this Issue Apr 16, 2012 · 6 comments

Comments

Projects
None yet
3 participants

nickray commented Apr 16, 2012

It seems sometimes you give currency in USD, and sometimes USD in currency (i.e., the inverse).

The example below is from my Python wrapper, it's RUB and CHF on 2012-01-03 and 2010-01-03.

In [36]: fxrates.alternative("RUB", date=120103)
RUB = 31.687000, CHF = 0.932150, base USD
Out[36]: 33.993455988842996

In [37]: fxrates.alternative("RUB", date=100103)
RUB = 0.032990, CHF = 1.035750, base USD
Out[37]: 0.03185131547188028

Owner

currencybot commented Apr 16, 2012

NB: skim-readers can skip to #30 (comment) for latest update!


Hey, I haven't heard of this happening before - all the open exchange rates
data is stored in JSON files though - so if you're able to pinpoint where
those inverse rates are coming from (which files), that would be very
helpful in figuring out whether something is wrong.

Let me know if you can find them! Thanks

Owner

currencybot commented Apr 16, 2012

Hey, thanks for that. I didn't notice you mentioned the dates before.

So that's a strange one, because many of the currencies seem accurate. The
2010 file is from our bulk import of historical data, which came from the
same source, but hence the issue where timestamp is a string (not integer
as it should be)

Thanks for bringing this up - I'll investigate this week and see if we need
to recrawl the historical data.

For what it's worth, are the latest values accurate? Most important right
now that the realtime exchange rates are OK.

Also - your python wrapper sounds legit, are you able to publish it
anywhere?

Cheers,

nickray commented Apr 16, 2012

I sent you a pull request for my Python wrapper, use at your own convenience.

The problem is that in fact the latest values are incorrect, at least for RUB and CHF.

Here's some data for RUB:
2011-11-05 00:00:00 0.03255
2011-11-06 00:00:00 0.03255
2011-11-07 00:00:00 0.03275
2011-11-08 00:00:00 31.4657
2011-11-09 00:00:00 0.03278
2011-11-10 00:00:00 0.03265
2011-11-11 00:00:00 0.03301
2011-11-12 00:00:00 0.03301
2011-11-13 00:00:00 0.03301
2011-11-14 00:00:00 0.03271
2011-11-15 00:00:00 30.75605
2011-11-16 00:00:00 30.76028811
2011-11-17 00:00:00 30.81582305
2011-11-18 00:00:00 30.93696145
2011-11-19 00:00:00 30.80805633
2011-11-20 00:00:00 30.86879322
2011-11-21 00:00:00 31.14126565

Your scraper script should definitely do some plausibility checks (e.g, is the number different than the last, has the price not moved by more than say twice the median daily price move).

Let me tell you, financial data is always dirty ;-). That's why the vendors can afford to charge ridiculous prices.

Cheers

Owner

currencybot commented Apr 16, 2012

Excellent info, thanks for the rundown on the dodgy data. I'll check out
the PR tomorrow, cheers for that.

Bumped this up to tomorrow's to-do list (it's bedtime here) and will get
back to you once I've investigated.

There are a lot of extra things the script needs to do, actually. Sanity
checks are a big one, but also cross-checking different providers for
discrepancy would be ideal. I have a lot of free time coming up so I'll be
implementing a bunch of those as separate processes (for now, they'll
simply flag for review when something looks odd).

Thanks again!

Collaborator

wjcrowcroft commented Apr 17, 2012

Hey, didn't realise I was writing from CurrencyBot's account! (was replying in Gmail) ... he won't be happy about that for sure.

I've checked out the latest files and looks like the realtime data is grabbing the correct values:

CHF: 0.91356,
RUB: 29.576,

CHF: compare 0.914101849 from https://www.google.com/search?q=1+USD+in+CHF
RUB: compare 29.577923 from https://www.google.com/search?q=1+USD+in+RUB

But you're right about the historical values (and implementing sanity checks nonetheless) so I'm leaving this one open until I can get the historical data figured out.

tl;dr for speed-readers:

latest.json rates and historical daily rates going back to 2011-11-16 are unaffected.

affected currencies: RUB and CHF (so far) in historical data before 2011-11-16

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment