Skip to content

tep-1015.adoc

Latest commit

 

History

History
512 lines (350 loc) · 15.2 KB

tep-1015.adoc

File metadata and controls

512 lines (350 loc) · 15.2 KB
Raw

TEP-1015: Price data support

Add price data support for currencies, shares and commodities.

Github issue: #36

Equity Export: #101

Summary

There are three different use cases for initial price data support:

  1. Current valuation of commodities, shares and currencies

  2. Historical valuation of commodities, shares and currencies, at certain point in time

  3. Simulation with historical price data. For example

    1. Historical purchasing power of money analysis

    2. Market value analysis of produced commodity (e.g. electricity)

Price Data lookup types

  1. Last recorded rate in PriceDB

  2. Before given timestamp

  3. At transaction time or nearest before the txn time

Fixed conversions

This TEP-1015 does not propose any method for calculating the fixed conversions (e.g. oz to g, gallons to litres, etc).

Inverted prices

At the moment inverted prices are not supported. With following PriceDB, conversion from USD to EUR is not supported.

PriceDB with no USD to EUR rate
P 2025-01-07 EUR 1.3 USD

For USD to EUR conversion, there must be an own entry for the inverted rate in the price db:

PriceDB with inverted rate
P 2025-01-07 EUR 1.3 USD
P 2025-01-07 USD 0.77 EUR ; 1/1.3 ~ 0.7692; Normally there is official inverted rate

Price conversion chains

Price conversions chains are not supported. For example, this conversion from EUR to USD via SEK and NOK is not supported:

PriceDB without direct EUR to USD rate
P 2025-01-07 EUR 11.50 SEK
P 2025-01-07 SEK  1.02 NOK
P 2025-01-07 NOK  0.09 USD ; EUR->SEK-NOK->USD = 1.055700; The real rate is 1.02

If this would be allowed, it would substantially complicate conversion and rate applying logic and could produce surprising results, especially when mixed with direct conversion rates.

Price Data application policy

last-price and given-time

  1. Used Price is based on rates noted in the pricedb. Possible costs recorded in the transactions are ignored.

  2. If there are multiple rates in the pricedb for the same commodity, the latest applicable is used (applicable by price lookup type).

txn-time

  1. Used Price is based on rates noted in the pricedb. Possible costs recorded in the transactions are ignored.

  2. The price rate, which has the same time as the transaction or the latest rate before transaction time, is used.

Historic price data with filters

Let’s assume there are following price data entries (three entries per year, distributed evenly):

P 2024-03-31 ACME 240331.00 USD
P 2024-06-30 ACME 240630.00 USD
P 2024-09-30 ACME 240930.00 USD
P 2024-12-31 ACME 241231.00 USD

Today is 2025-01-01 and we run filtering query which will include txns from (inclusive - exclusive) range [ 2024-01-01 .. 2024-04-01 [ into our reports.

If we like to use the same conversion rate, which was used when the 2024/Q1 report was generated, the price data lookup type should be 2. Before given timestamp with the same value as filter’s end time --report.price-before=2024-04-01.

The used price data for reporting would be 240331.00 USD, (the P 2024-03-31 ACME 240331.00 USD entry), which is the nearest earlier recorded price in the Price DB. This will ensure that reported values are same, regardless if the report is run on 2024-04-01 or 2025-01-01.

There is future option to do this automatically, see Filtering Changes.

Journal File Format

No change would be made to the journal file format. A new file type would be added with a format similar to the default format of Ledger CLI as described here.

'P' TIMESTAMP COMMODITY AMOUNT COMMODITY OPT_COMMENT

Where OPT_COMMENT is prefixed with ';' space+, this is similar as Txn comments.

For example:

P 2025-01-07 EUR 1.0393 USD ; as reported by Central Bank of Europe

There are some existing usage for this format:

Timestamp handling

The timestamp will use the same logic as transaction timestamps:

  • RFC3339 timestamp with offset information

  • Timestamp without zone or offset, then the kernel.timestamp.timezone value is used as zone/offset.

  • Date, then kernel.timestamp.default-time is used with kernel.timestamp.timezone.

The resulting timestamp is always with offset.

Implementation

CLI Changes

New command line switch to activate price data, this could be implemented later.

--price.db-path

Path to the price data file

--price.lookup-type

Price lookup type, valid values are:

  • none

  • last-price

  • txn-time

  • given-time

--price.before <ts>

Date or timestamp to be used in price lookup, see also Future Plans and Postponed (PP) Features.

  • This is valid option only with last-price or given-time price lookup types.

  • This is exclusive time, e.g. the first price db entry before this time is used.

  • If ts is date, then kernel.timestamp.default-time and kernel.timestamp.timezone is used for timestamp.

  • If ts has date and time, but it’s missing offset/zone information, then kernel.timestamp.timezone is used.

--report.commodity

Commodity to be used in reports

CONF Changes

New configuration keys for pricedb functionality:

[price]
### this is similar as `accounts`, `commodidities` and `tags`.
### "none" means that no price data is loaded.
### In that case also `price.lookup-type` must be "none".
###
### To temporaryly disable price data functionality,
### use CONF and CLI switch `price.lookup-type  = "none"`
###
db-path = "path/to/price.db"
###
### Price lookup type
###
### Valid values are:
###   - "none"
###   - "last-price"
###   - "txn-time"
###   - "given-time", with this CLI `--price.before` must be used
###
lookup-type = "txn-time"

[report]
###
### Target commodity in reports
###
commodity = "TCKLR"

Filtering Changes

Future change:

Add extra attribute to TxnFilter definition end_ts, to define which price data will be used with reports when using filtering and restricting reporting to past dates.

Machinery

Changes to machinery

  • ❏ Add a new model PriceEntry with fields

    • timestamp with offset

    • source commodity (with inferred amount 1)

    • target commodity with rate

    • optional comment

The report.report-timezone is used to convert price data to displayed zone in the reports.

API Changes

Api changes to server or client interfaces.

  • ❏ Future: end_ts attribute in tackler_api::filters::FilterDefinition

JSON Model

Changes to JSON model

Reporting price and commodity conversion related data in:

  • ❏ Metadata

  • ❏ JSON Reports

    • ❏ BalanceReport

    • ❏ BalanceGroupReport

    • ❏ RegisterReport

New Dependencies

No new dependencies

Reporting

Changes to reports or reporting

The used prices are reported as part of the Metadata section of the report, when single rate for particular conversion is used. In case of txn-time, variable cost bases will be noted in the metadata.

Future: This could be behind a switch in the future, so that this can be turned off, especially in the case that there are many commodities / prices to be reported.

Git Storage
         commit : 4aa4e9797501c1aefc92f32dff30ab462dae5545
      reference : txns-1E1
      directory : txns
         suffix : .txn
        message : txns-1E1: 2016/12

Txn Set Checksum
        SHA-256 : 9b29071e1bf228cfbd31ca2b8e7263212e4b86e51cfee1e8002c9b795ab03f76
       Set size : 10

Price Data
           time : 2025-01-08 12:13:14
      commodity : EUR
          value : 1.234 USD
                -
           time : 2024-12-31 08:00:00
      commodity : He·bar_50L·tank
          value : 3.45 EUR

Balance Report

Changes to balance report

  • ❏ item

Balance Group Report

Sub-Group price data

This implementation uses the same logic as Balance Report, e.g. single price value is used for all sub-groups with last-price and given-time price lookup types. There could be an conf/cli option to change this in the future, so that each sub-group could have their own price data based on time to used to greate that group.

Register Report

Changes to register report

Register report with price data with fixed time based price
Price Data
           time : 2024-01-31 00:00:00
      commodity : aaa
          value : 31.001 TCKLR

REGISTER
--------
2024-01-01
            e:conv                        1.00 aaa                        31.001 TCKLR
--------------------------------------------------------------------------------------
2024-01-12
            e:conv                        1.00 aaa                        62.002 TCKLR
--------------------------------------------------------------------------------------
2024-01-24
            e:conv                        1.00 aaa                        93.003 TCKLR
--------------------------------------------------------------------------------------
Register report with variable price data
Price Data
           time : transaction time
      commodity : aaa
          value : - TCKLR

REGISTER
--------
2024-01-01
            e:conv                        1.00 aaa @ 1.001 TCKLR           1.001 TCKLR
--------------------------------------------------------------------------------------
2024-01-12
            e:conv                        1.00 aaa @ 12.001 TCKLR         13.002 TCKLR
--------------------------------------------------------------------------------------
2024-01-24
            e:conv                        1.00 aaa @ 24.001 TCKLR         37.003 TCKLR
--------------------------------------------------------------------------------------

Exporting

Changes to exports or exporting

Equity Export

If Price conversion is activate with Equity Export, create equity transactions with converted prices and include rate information into Equity Transaction’s comment.

Identity Export

No changes to Identity Export

Documentation

  • ❏ ./readme.adoc: Update TEP index

  • ❏ ../../README.adoc: is it a new noteworthy feature?

  • ❏ ../../CHANGELOG: add new item

  • ❏ Does it warrant own T3DB file?

  • ❏ User docs

    • ❏ User Manual

      • ❏ cli-arguments

        • ✓ --price.db-path

        • ✓ --price.lookup-type

        • ✓ --price.before

        • ❏ --report.commodity

    • ❏ tackler.toml

      • ✓ price.db-path

      • ✓ price.lookup-type

      • ❏ report.commodity

    • ❏ examples

  • ❏ Developer docs

    • ❏ API changes

      • ❏ Server API changes

      • ❏ Client API changes

      • ❏ JSON Examples

Future Plans and Postponed (PP) Features

How and where to go from here?

  • Maybe support for price conversion chains? (EUR → SEK → NOK → USD)

  • Fixed, constant conversions (e.g. oz to g, gallons to litres, etc)

  • More options to select used price data based on time:

    • --price.after

    • --price.at (exact time, with --price.resolution)

    • --price.resolution (e.g. 1h, 1d, 1w, 1m, 1y)

  • Inferred prices by recorded costs in transactions

    • Option for txn-time to use inferred prices

Used Price date

CLI/CONF option to define machine how price data will be used with reports when using filtering and restricting reporting to past dates, see also Filtering Changes.

Possible idea-level options could be:

  • Latest recorded value in PriceDB

  • Latest available price before given filter or timestamp (e.g. --report.price-before=2024-05-01)

  • Next, nearest future price

  • Linear fitting between nearest past and nearest future prices

Postponed (PP) Features

Anything which wasn’t implemented?

Tests

Normal, ok-case tests to validate functionality:

  • ✓ strict-mode

    • ✓ On

    • ✓ Off (without commodities)

  • ✓ txn with empty commodities when price conv is activated

  • ✓ No conversion

    • ✓ commodity but no conversion activated

    • ✓ Secondary account with non-conv commodity (val-pos notation)

  • ✓ Selecting correct price by time

    • ✓ at-txn

      • ✓ too early price

      • ✓ too late price

    • ✓ last-price

      • ✓ too early price

      • [-] too late price

    • ✓ CLI given-time

      • ✓ too early price

      • ✓ too late price

      • ✓ timestamp is after last filtered txn ts and after last price

    • [-] Filter: Txn-TS-END

      • [-] too early price

      • [-] too late price

      • [-] filter ts is after last filtered txn ts

  • ✓ Multiple sources to same target

    • ✓ {aaa, bbb, ccc} → TCKLR

  • ✓ postings with value positions

    • ✓ value position for source posting

    • ✓ value position for target posting

  • ✓ Balance-Group

    • ✓ Commodities distributed so that each Balance-Group has different conversion

      • ✓ Conv: a → TCKLR, b→TCKLR, etc.

    • [-] Each Balance-Group have different conversion value

      • [-] filter: group-selector == txn-ts-end (Is this desired behaviour?)

  • ✓ Metadata testing

    • ✓ Fixed conv rate, multiple commodities, different ts

    • ✓ Timed conv rate, multiple commodities

    • ✓ Multiple possible conversions, but only one real conv (filter txn by comm)

    • Timestamps

      • ✓ Date resolution

      • ✓ Second resolution

      • ✓ Nanosecond resolution

  • ✓ PriceDB time handling

    • ✓ with different time resolutions

      • ✓ Date resolution

      • ✓ Second resolution

      • ✓ Nanosecond resolution

    • ✓ Missing offset

      • ✓ Date resolution

      • ✓ Second resolution

  • ✓ Target commodity not found (at all) in pricedb

  • ✓ Transaction commodity is same as target commodity (recorded cost / unit price in target commodity)

Errors

Various error cases:

  • ✓ e: no source to target conversion in pricedb

  • ✓ e: TS with LookupType != GivenTime

  • ✓ e: GivenTime, no TS

Perf

Is there need to run or create new perf tests?

  • ❏ Old tests, no conversion

    • ❏ Old tests, activated conversion, no match

  • ❏ New perf data with commodity

    • ❏ Perf with conversion

  • ❏ Perf benchmark (cargo bench) for PriceDB + Cache

    • ❏ 10, 100, 1000, 10_000 commodities (ECB publish rates for 37 currencies, NYSE has ~3k companies)

Feature and Test Coverage Tracking

Feature-id

  • id: 98c2b696-d250-4141-bd82-c4126ec11c1d

  • subject: "Price data support"

Feature-id

  • id:4d6f0dac-c202-45f4-9b95-7682d6c8df94

  • parent: 98c2b696-d250-4141-bd82-c4126ec11c1d

  • subject: "Price conversions"

Metadata template for Feature and Test Coverage Tracking

Tackler is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.