| title | order | status |
|---|---|---|
Dataset |
2 |
published |
Datasets refer to a table like data with typed columns. Datasets can be
sourced from external datasets or derived from other
datasets via pipelines. Datasets are written as
Pydantic inspired Python classes decorated
with the @dataset decorator. Let's look at an example:
A dataset has few fields (interchangeably referred to as columns throughout the documentation) with types and unique names. Each field must have has a pre-specified datatype. See the typing section to learn the types supported by Fennel.
You might have noticed the field(...) descriptor next to uid and
update_time fields. These optional descriptors are used to provide non-typing
related information about the field. Currently, Fennel supports two
field descriptors:
Key Fields
Key fields are those with field(key=True) set on them. The semantics of this
are somewhat similar to those of primary key in relational datasets. Datasets
can be looked-up by providing the value of key fields. It is okay to have a
dataset with zero key fields (e.g. click streams) - in those cases, it's not
possible to do lookups on the dataset at all. Fennel also supports having
multiple key fields on a dataset - in that case, all of those need to be
provided while doing a lookup. Field with optional data types can not be key
fields.
Timestamp Field
Timestamp fields are those with field(timestamp=True) set on them. Every
dataset must have exactly one timestamp field and this field should always
be of type datetime. Semantically, this field must represent 'event time'
of the row. Fennel uses timestamp field to associate a particular state of
a row with a timestamp. This allows Fennel to handle out of order events,
do time window aggregations, and compute point in time correct features for
training data generation.
If a dataset has exactly one field of datetime type, it is assumed to be
the timestamp field of the dataset without explicit annotation. However, if
a dataset has multiple timestamp fields, one of those needs to be explicitly
annotated to be the timestamp field.
Here are some examples of valid and invalid datasets:
Keyed datasets can be indexed by setting index to be True in the dataset
decorator and indexed datasets can be looked up via the lookup method on
datasets.
By default, setting index=True builds both the online and full historical
offline indices. That can be configured by setting online and offline
kwargs on the decorator (the default configuration is online set to True and
offline set to forever).
Selectively turning only one of these on can be a good way to save storage and
compute costs. A dataset needs to be indexed with offline set to forever for
it to be joinable as the RHS dataset.
All Fennel datasets are versioned and each version is immutable. The version of
a dataset can be explicitly specified in the @dataset decorator as follows:
Increasing the version of a dataset can be accompanied with any other changes - changing schema, changing source, change pipeline code, adding/removing index etc. In either scenario, Fennel recomputes the full dataset when the version is incremented.
The version of a dataset also captures all its ancestry graph. In other words, when the version of a dataset is incremented, the version of all downstream datasets that depend on it must also be incremented, leading to their reconstruction as well.
As of right now, Fennel doesn't support keeping two versions of a dataset alive simultaneously and recommends to either create datasets with different names or run two parallel branches with different versions of the dataset.
Datasets can be annotated with useful meta information via metaflags - either at the dataset level or at the single field level. To ensure code ownership, Fennel requires every dataset to have an owner. Here is an example:
Typing the same owner name again and again for each dataset can get somewhat
repetitive. To prevent that, you can also specify an owner at the module level
by specifying __owner__ and all the datasets in that module inherit
this owner. Example:
In this example, datasets UserBMI and UserLocation both inherit the owner
from the module level __owner__ whereas dataset UserName overrides it by
providing an explicit meta flag.