Model data as beliefs (at a certain time) about events (at a certain time).
timely-beliefs package provides a convenient data model for numerical time series,
that is both simple enough for humans to understand and sufficiently rich for forecasting and machine learning.
The data model is an extended pandas DataFrame that assigns properties and index levels to describe:
- What the data is about
- Who (or what) created the data
- When the data was created
- How certain they were
The package contains the following functionality:
- A model for time series data, suitable for a notebook or a database-backed program (using sqlalchemy)
- Selecting/querying beliefs, e.g. those held at a certain moment in time
- Computing accuracy, e.g. against after-the-fact knowledge, also works with probabilistic forecasts
- Resampling time series with uncertainty (experimental)
- Visualising time series and accuracy metrics (experimental)
Some use cases of the package:
- Clearly distinguish forecasts from rolling forecasts.
- Analyse your predictive power by showing forecast accuracy as you approach an event.
- Learn when someone is a bad predictor.
- Evaluate the risk of being wrong about an event.
Check out our interactive demonstration comparing forecasting models for renewable energy production. These visuals are created simply by calling the plot method on our BeliefsDataFrame, using the visualisation library Altair.
Table of contents
- The data model
- Database storage
- More examples
The data model
The BeliefsDataFrame is the basic data model that represents data as probabilistic beliefs about events. It is an extended pandas DataFrame with the following index levels:
event_start; keeping track of the time of whatever it is that the data point describes (an event)
belief_time; keeping track of the time at which the data point was created (a belief)
source; keeping track of who or what created the data point (a source)
cumulative_probability; keeping track of the confidence in the data point (a probability)
Together these index levels describe data points as probabilistic beliefs. Because of the sparse representation of index levels (a clever default setting in pandas) we get clean-looking data, as we show here in a printout of the example BeliefsDataFrame in our examples module:
>>> import timely_beliefs >>> df = timely_beliefs.examples.example_df >>> df.head(8) event_value event_start belief_time source cumulative_probability 2000-01-03 09:00:00+00:00 2000-01-01 00:00:00+00:00 Source A 0.1587 90 0.5000 100 0.8413 110 Source B 0.5000 0 1.0000 100 2000-01-01 01:00:00+00:00 Source A 0.1587 99 0.5000 100 0.8413 101
The first 8 entries of this BeliefsDataFrame show beliefs about a single event. Beliefs were formed by two distinct sources (A and B), with the first updating its beliefs at a later time. Source A first thought the value of this event would be 100 ± 10 (the probabilities suggest a normal distribution), and then increased its accuracy by lowering the standard deviation to 1. Source B thought the value would be equally likely to be 0 or 100.
More information about what actually constitutes an event is stored as metadata in the BeliefsDataFrame. The sensor property keeps track of invariable information such as the unit of the data and the resolution of events.
>>> df.sensor <Sensor: Sensor 1>
Currently a BeliefsDataFrame contains data about a single sensor only. For a future release we are considering adding the sensor as another index level, to offer out-of-the-box support for aggregating over multiple sensors.
- Read more about how the DataFrame is keeping track of time.
- Read more about how the DataFrame is keeping track of confidence.
- Discover convenient slicing methods, e.g. to show a rolling horizon forecast.
- Serve your data fast by resampling, while taking into account auto-correlation.
- Track where your data comes from, by following its lineage.
Keeping track of time
Events and sensors
Numerical data quantifies something, it is a value assigned to an event;
a physical event, such as a temperature reading, power flow or water flow,
or an economical event, such as daily goods sold or a price agreement about a future delivery.
An event typically takes some time, so it has an
event_start and an
To observe an event, we need a sensor. We say that a sensor has a certain
event_resolution = event_end - event_start
We define the resolution to be a fixed property of the sensor. For example:
- An anemometer (wind speed meter) determines the number of revolutions within some period of time.
- A futures contract determines the price of a future delivery within some period of time.
Beliefs in physics
Assigning a value to an event implies a belief.
We say that a belief can be formed some time before or after an event, and call this time
A weather forecast is a good example, where:
belief_time < event_end
For physical events, the time at which we can say the event could be known (which we call
knowledge_time) is at the
knowledge_time = event_end
The forecast horizon, or
belief_horizon, says how long before (the event could be known) the belief was formed:
belief_horizon = knowledge_time - belief_time
For example, a forecast of solar irradiation on June 10th 2017 with a horizon of 27 hours means a belief time of 9 PM on June 9th 2017. That is:
event_start = datetime(2017, 6, 10, hour=0) event_end = datetime(2017, 6, 11, hour=0) belief_horizon = timedelta(hours=27) belief_time = datetime(2017, 6, 9, hour=21)
Beliefs in economics
For economical events, the time at which we say an event could be known is typically not at the
Most contracts deal with future events, such that:
knowledge_time < event_start
knowledge_horizon says how long before (the event starts) the event could be known:
knowledge_horizon > 0 # for most economical events knowledge_horizon = -resolution # for physical events
We define the knowledge horizon to be a fixed property of the sensor. For example, hourly prices on the day-ahead electricity market are determined at noon one day before delivery starts, such that:
knowledge_time = event_start.replace(hour=12) - timedelta(days=1)
Then for an hourly price between 3 and 4 PM on June 10th 2017:
event_start = datetime(2017, 6, 10, hour=15) event_end = datetime(2017, 6, 10, hour=16) knowledge_time = datetime(2017, 6, 9, hour=12) knowledge_horizon = timedelta(hours=27)
Continuing this example, a price forecast with a forecast horizon of 1 hour constitutes a belief formed at 11 AM:
belief_horizon = timedelta(hours=1) belief_time = datetime(2017, 6, 9, hour=11)
In general, we have the following relationships:
belief_time + belief_horizon = knowledge_time belief_time + belief_horizon + knowledge_horizon = event_start belief_time + belief_horizon + knowledge_horizon + event_resolution = event_end
A common misconception
In many applications, people tend to interpret a forecast horizon as the duration between forming the belief and the start of the event. When this happens, we have:
forecast_horizon = event_start - belief_time
forecast_horizon = belief_horizon + knowledge_horizon
For example, consider a forecast formed at 9 AM about the average wind speed between 10 and 11 AM. It may feel intuitive to talk about a forecast horizon of 1 hour, because people tend to index events by their start time and then talk about the timing of beliefs with respect to that index.
While this is a perfectly acceptable definition, we set out to be precise in handling the timing of beliefs.
Therefore, we use the term
belief_horizon rather than
forecast_horizon throughout the
Instantaneous events can be modelled by defining a sensor with:
event_resolution = 0
Beliefs about past events can be modelled using a negative horizon:
belief_horizon < 0 knowledge_time < belief_time
That is, your beliefs can still change after you (think you) know about an event. NB in the following case a price has been determined (you could know about it) for a future event:
knowledge_time < belief_time < event_start
Our concept of
knowledge_time supports to define sensors for agreements about ongoing or past events, such as ex-post contracts.
event_start < knowledge_time -resolution < knowledge_horizon < 0 # for ongoing events knowledge_horizon < -resolution # for past events
Keeping track of confidence
To keep track of the confidence of a data point, timely-beliefs works with probability distributions.
More specifically, the BeliefsDataFrame contains points of interest on the cumulative distribution function (CDF),
and leaves it to the user to set an interpolation policy between those points.
This allows you to describe both discrete possible event values (as a probability mass function) and continuous possible event values (as a probability density function).
A point of interest on the CDF is described by the
cumulative_probability index level (ranging between 0 and 1) and the
event_value column (a possible value).
The default interpolation policy is to interpret the CDF points as discrete possible event values, leading to a non-decreasing step function as the CDF. In case an event value with a cumulative probability of 1 is missing, the last step is extended to 1 (i.e. the chance of an event value that is greater than largest available event value is taken to be 0).
A deterministic belief consists of a single row in the BeliefsDataFrame. Regardless of the cumulative probability actually listed (we take 0.5 by default), the default interpolation policy will interpret the single CDF point as an event value stated with 100% certainty. The reason why we choose a default cumulative probability of 0.5 instead of 1 is that, in our experience, sources more commonly intend to report their expected value rather than an event value with absolute confidence.
A probabilistic belief consists of multiple rows in the BeliefsDataFrame,
with a shared
source, but different
For a future release we are considering adding interpolation policies to interpret the CDF points as describing a normal distribution or a (piecewise) uniform distribution,
to offer out-of-the-box support for resampling continuous probabilistic beliefs.
Convenient slicing methods
Being an extension of the pandas DataFrame, all of pandas excellent slicing methods are available on the BeliefsDataFrame. For example, to select all beliefs about events from 11 AM onwards:
>>> from datetime import datetime, timedelta >>> from pytz import utc >>> df = timely_beliefs.examples.example_df >>> df[df.index.get_level_values("event_start") >= datetime(2000, 1, 3, 11, tzinfo=utc)]
timely-beliefs provides custom methods to conveniently slice through time in different ways.
Select the latest forecasts from a rolling viewpoint (beliefs formed at least 2 days and 10 hours before the event could be known):
>>> df.rolling_viewpoint(timedelta(days=2, hours=10)) event_value event_start belief_horizon source cumulative_probability 2000-01-03 10:00:00+00:00 2 days 10:15:00 Source A 0.1587 180 0.5000 200 0.8413 220 Source B 0.5000 0 1.0000 200 2000-01-03 11:00:00+00:00 2 days 10:15:00 Source A 0.1587 297 0.5000 300 0.8413 303 Source B 0.5000 0 1.0000 300 2000-01-03 12:00:00+00:00 2 days 11:15:00 Source A 0.1587 396 0.5000 400
Select the latest forecasts from a fixed viewpoint (beliefs formed at least before 2 AM January 1st 2000:
>>> df.fixed_viewpoint(datetime(2000, 1, 1, 2, tzinfo=utc)).head(8) event_value event_start belief_time source cumulative_probability 2000-01-03 09:00:00+00:00 2000-01-01 01:00:00+00:00 Source A 0.1587 99 0.5000 100 0.8413 101 Source B 0.5000 0 1.0000 100 2000-01-03 10:00:00+00:00 2000-01-01 01:00:00+00:00 Source A 0.1587 198 0.5000 200 0.8413 202
Select a history of beliefs about a single event:
>>> df.belief_history(datetime(2000, 1, 3, 11, tzinfo=utc)) event_value belief_time source cumulative_probability 2000-01-01 00:00:00+00:00 Source A 0.1587 270 0.5000 300 0.8413 330 Source B 0.5000 0 1.0000 300 2000-01-01 01:00:00+00:00 Source A 0.1587 297 0.5000 300 0.8413 303 Source B 0.5000 0 1.0000 300
Upsample to events with a resolution of 5 minutes:
>>> from datetime import timedelta >>> df = timely_beliefs.examples.example_df >>> df5m = df.resample_events(timedelta(minutes=5)) >>> df5m.sort_index(level=["belief_time", "source"]).head(9) event_value event_start belief_time source cumulative_probability 2000-01-03 09:00:00+00:00 2000-01-01 00:00:00+00:00 Source A 0.1587 90.0 0.5000 100.0 0.8413 110.0 2000-01-03 09:05:00+00:00 2000-01-01 00:00:00+00:00 Source A 0.1587 90.0 0.5000 100.0 0.8413 110.0 2000-01-03 09:10:00+00:00 2000-01-01 00:00:00+00:00 Source A 0.1587 90.0 0.5000 100.0 0.8413 110.0
Downsample to events with a resolution of 2 hours:
>>> df2h = df.resample_events(timedelta(hours=2)) >>> df2h.sort_index(level=["belief_time", "source"]).head(15) event_value event_start belief_time source cumulative_probability 2000-01-03 09:00:00+00:00 2000-01-01 00:00:00+00:00 Source A 0.158700 90.0 0.500000 100.0 1.000000 110.0 2000-01-03 10:00:00+00:00 2000-01-01 00:00:00+00:00 Source A 0.025186 225.0 0.079350 235.0 0.133514 240.0 0.212864 245.0 0.329350 250.0 0.408700 255.0 0.579350 260.0 0.750000 265.0 1.000000 275.0 2000-01-03 12:00:00+00:00 2000-01-01 00:00:00+00:00 Source A 0.158700 360.0 0.500000 400.0 1.000000 440.0
Notice the time-aggregation of probabilistic beliefs about the two events between 10 AM and noon. Three possible outcomes for both events led to nine possible worlds, because downsampling assumes by default that the values indicate discrete possible outcomes.
Get the (number of) sources contributing to the BeliefsDataFrame:
>>> df.lineage.sources array([<BeliefSource Source A>, <BeliefSource Source B>], dtype=object) >>> df.lineage.number_of_sources 2
All of the above can be done with
TimedBelief objects in a
BeliefsDataFrame. However, if you are dealing with a lot of data and need performance, you'll want to persist your belief data in a database. The timely-beliefs library supports this, as all relevant classes have a subclass which also derives from sqlalchemy's declarative base.
The timely-beliefs library comes with database-backed classes for the three main components of the data model -
DBBeliefSource. Objects from these classes can be used just like their super classes, so for instance
DBTimedBelief objects can be used for creating a
Table creation and storage
You can let sqlalchemy create the tables in your database session and start using the DB classes (or subclasses, see below) and program code without much work by yourself. The database session is under your control - where or how you get it, depends on the context you're working in. Here is an example how to set up a session and also have sqlachemy create the tables:
from timely_beliefs.db_base import Base as TBBase from sqlalchemy.orm import sessionmaker SessionClass = sessionmaker() session = None def create_db_and_session(): engine = create_engine("your-db-connection-string") SessionClass.configure(bind=engine) TBBase.metadata.create_all(engine) if session is None: session = SessionClass() # maybe add some inital sensors and sources to your session here ... return session
Note how we're using timely-belief's sqlalchemy base (we're calling it
TBBase) to create them. This does not create other tables you might have in your data model.
DBBeliefSource also can be subclassed, if more attributes are needed. This should be most interesting for sensors and maybe belief sources.
Below is an example, for the case of a db-backed case, where we wanted a sensor to have a location. We added three attributes,
from timely_beliefs import DBSensor from sqlalchemy import Column, Float, String class DBLocatedSensor(DBSensor): """A sensor with a location lat/long and location name""" latitude = Column(Float(), nullable=False) longitude = Column(Float(), nullable=False) location_name = Column(String(80), nullable=False) def __init__( self, latitude: float = None, longitude: float = None, location_name: str = "", **kwargs, ): self.latitude = latitude self.longitude = longitude self.location_name = location_name DBSensor.__init__(self, **kwargs)
The accuracy of a belief is defined with respect to some reference. The default reference is the most recent belief held by the same source, but it is possible to set beliefs held by a specific source at a specific time to serve as the reference instead.
Accuracy and error metrics
To our knowledge, there is no standard metric for accuracy.
However, there are some standard metrics for what can be considered to be its opposite: error.
By default, we give back the Mean Absolute Error (MAE),
the Mean Absolute Percentage Error (MAPE)
and the Weighted Absolute Percentage Error (WAPE).
Each of these metrics is a representation of how wrong a belief is (believed to be),
with its convenience depending on use case.
For example, for intermittent demand time series (i.e. sparse data with lots of zero values) MAPE is not a useful metric.
For an intuitive representation of accuracy that works in many cases, we suggest to use
df["accuracy"] = 1 - df["wape"].
With this definition:
- 100% accuracy denotes that all values are correct
- 50% accuracy denotes that, on average, the values are wrong by half of the reference value
- 0% accuracy denotes that, on average, the values are wrong by exactly the reference value (i.e. zeros or twice the reference value)
- negative accuracy denotes that, on average, the values are off-the-chart wrong (by more than the reference value itself)
The previous metrics (MAE, MAPE and WAPE) are technically not defined for probabilistic beliefs.
However, there is a straightforward generalisation of MAE called the Continuous Ranked Probability Score (CRPS), which is used instead.
The other metrics follow by dividing over the deterministic reference value.
For simplicity in usage of the
the metrics names in the BeliefsDataFrame are the same regardless of whether the beliefs are deterministic or probabilistic.
It is possible that the reference itself is a probabilistic belief rather than a deterministic belief. Our implementation of CRPS handles this case, too, by calculating the distance between the cumulative distribution functions of each forecast and reference (Hans Hersbach, 2000). As the denominator for calculating MAPE and WAPE, we use the expected value of the probabilistic reference.
There are two common use cases for wanting to know the accuracy of beliefs,
each with a different viewpoint.
With a rolling viewpoint, you get the accuracy of beliefs at a certain
belief_horizon before (or after)
for example, some days before each event ends.
>>> df.rolling_viewpoint_accuracy(timedelta(days=2, hours=9), reference_source=df.lineage.sources) mae mape wape source Source A 1.482075 0.014821 0.005928 Source B 125.853250 0.503413 0.503413
With a fixed viewpoint, you get the accuracy of beliefs held at a certain
>>> df.fixed_viewpoint_accuracy(datetime(2000, 1, 2, tzinfo=utc), reference_source=df.lineage.sources) mae mape wape source Source A 0.00000 0.000000 0.000000 Source B 125.85325 0.503413 0.503413
Create interactive charts using Altair and view them in your browser.
>>> chart = df.plot(reference_source=df.lineage.sources, show_accuracy=True) >>> chart.serve()
This will create an interactive Vega-Lite chart (Satyanarayan et al., 2016) like the one in the screenshot at the top of this Readme. At this time, we chose to show the (possibly more intuitive) forecast horizon for visualisation, rather than our more precise definition of belief horizon.
How to interact with the chart
Select a time window
Click and drag in
Select time window to zoom in on a subset of your data.
You can then drag your selected time window to move through time.
Reset with a double click.
Select a belief time
Travel through time as you select a belief time by clicking in
Reset with a double click.
Select a horizon
Look only at what was believed some duration before each event by clicking in
Select forecasting horizon.
Double click to select all horizons.
The chart allows you to switch between a fixed and rolling viewpoint as follows:
- Fixed viewpoint: first select all horizons by double clicking in
Select forecasting horizon, then click anywhere in
Model resultsto travel through time.
- Rolling viewpoint: first click somewhere on the far right in
Model resultsto look back at the full range of your data, then select a specific horizon by clicking in
Select forecasting horizon.
- Hans Hersbach. Decomposition of the Continuous Ranked Probability Score for Ensemble Prediction Systems in Weather and Forecasting, Volume 15, No. 5, pages 559-570, 2000.
- Arvind Satyanarayan, Dominik Moritz, Kanit Wongsuphasawat, and Jeffrey Heer. Vega-Lite: A Grammar of Interactive Graphics in IEEE transactions on visualization and computer graphics, Volume 23, No. 1, pages 341-350, 2016.