Handling of historic currency/stock prices

[edwintorok]

Is your feature request related to a problem? Please describe.

I want to import historic currency exchange rates and stock prices into hledger-flow.
Unfortunately using a construct script doesn't work because hledger-flow filters out and only retains transactions, not prices, see

hledger-flow/src/Hledger/Flow/CSVImport.hs

Line 112 in 14e0150

let args = ["print", "--rules-file", format fp rf, "--file", format fp csvSrc, "--output-file", format fp journalOut]

Version and Runtime Information

$ hledger-flow --version
hledger-flow 0.12.2.99 linux x86_64 ghc 8.6

Is this the latest version?

No

Our Example Statements Repository

FYI, we have a repo with some example transactions which you can use to run hledger-flow on:
https://github.com/apauley/hledger-flow-example

Can you give examples of what you would like by running hledger-flow on these files?

No, they do not contain prices.

Describe the solution you'd like
It should be possible to import prices, not just transactions.
First step would be to have an equivalent of the construct script that can output only prices in a format that hledger understands.
A second step would be to have support for csv import rules that can deal with the various date formats and convert to hledger format.

Describe alternatives you've considered

I wrote a script that generates .journal files and puts them into a _manual_ folder.
This is a bit of a hack, because it is not part of the usual hledger-flow import, I have to run it separetely.
What I ended up having is this:

cat import/edwin/_manual_/run.sh
#/bin/sh
set -eu
cd prices;
(for i in $(ls -1d */ | cut -f1 -d/); do
    for j in ${i}/1in/*/*; do
        ${i}/construct "${j}" - X "${i}" X
    done
done) >pricelist
cd ..
for i in 2017 2018 2019; do
    echo 'D £1,000.00' >${i}/pre-import.journal
    grep $i prices/pricelist | sed -e 's/"EURGBP"/€/' >>${i}/pre-import.journal
done

cat import/edwin/_manual_/prices/construct
#!/bin/sh
# this is very slow, should reimplement in python or something else
set -eu
IFS=^
cut -f1-2 --output-delimiter=^ <"$1" | while read D PRICE; do
    D=$(date +%F -d "$D")
    echo P $D \"$4\" $PRICE
done

tail import/edwin/_manual_/prices/EURGBP/1in/2019/EURGBP
Jan 11, 2019 	0.8925 	0.9020 	0.9064 	0.8921 	-1.05%
Jan 10, 2019 	0.9020 	0.9026 	0.9063 	0.9011 	-0.06%
Jan 09, 2019 	0.9025 	0.8997 	0.9044 	0.8973 	0.33%
Jan 08, 2019 	0.8995 	0.8980 	0.9010 	0.8949 	0.18%
Jan 07, 2019 	0.8979 	0.8951 	0.8995 	0.8948 	0.20%
Jan 06, 2019 	0.8961 	0.8952 	0.8967 	0.8949 	0.09%
Jan 04, 2019 	0.8953 	0.9019 	0.9032 	0.8942 	-0.75%
Jan 03, 2019 	0.9021 	0.9000 	0.9133 	0.8986 	0.28%
Jan 02, 2019 	0.8996 	0.8991 	0.9034 	0.8981 	0.07%
Jan 01, 2019 	0.8990 	0.8985 	0.9057 	0.8953 	0.00%

Additional context

Obtaining the prices themselves is another matter, but that is independent of hledger-flow.
What I did was just copy/paste the tables from here, but this could easily be automated by something that fetches the webpage and parses it.

For example investment stock prices can be fetched from (using a random example), you can view 1 year ranges:
https://markets.ft.com/data/funds/tearsheet/historical?s=GB0033053710:GBX

Currencies can be found in a similar place, see:
https://github.com/barrucadu/hledger-scripts/blob/master/market-prices/market-prices.py

Stock prices can be found here (again using a random example)
https://uk.finance.yahoo.com/quote/GOOG/history/?guccounter=1

I haven't done anything interesting with the prices yet, other than being able to see how much I own were I to convert everything into GBP right now, so having daily price granularity is probably an overkill, but it was the easiest way to obtain the data.

[apauley]

@edwintorok I know it has been a while since you posted this, but I have finally implemented something (released in v 0.13.1) that can deal with prices.

In the off chance that you still find this useful, please have a look:
https://github.com/apauley/hledger-flow/#price-files