Add end_value support to incremental

[steinitzu]

Implements chunked loading support described in dlt-hub/verified-sources#197 (comment)

Usage e.g.

import dlt
from dlt.common import pendulum


@dlt.resource
def some_data(
    updated: dlt.sources.incremental[pendulum.DateTime] = dlt.sources.incremental(
        'updated', initial_value=pendulum.DateTime(2000, 1, 1)
    )
):
    ...


# Backfill a time range

some_data(updated=dlt.sources.incremental(
    initial_value=pendulum.DateTime(2023, 1, 1), end_value=pendulum.DateTime(2023, 1, 2)
))

When end_value is reached the generator is stopped and incremental state gets updated with either end_value or previous last_value (whichever is higher (or lower, etc, depending on last_value_func)).
So it's possible to load chunks in any order and then stop sending end_value arg to continue incremental load and vice versa.

Decided on using initial_value instead of adding another "start" argument. When end_value is supplied, the initial acts as an override.

[netlify]

✅ Deploy Preview for dlt-hub-docs ready!

Name	Link
🔨 Latest commit	5f31a42
🔍 Latest deploy log	https://app.netlify.com/sites/dlt-hub-docs/deploys/64a9955c5671ce000799a60a
😎 Deploy Preview	https://deploy-preview-467--dlt-hub-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[rudolfix]

PR looks good but the whole concept raises some questions:

1. when having a concrete chunk to load we should IMO not modify the state at all. we may read the state to get ie. last_value or initial_value but I think it's better if we require initial value to be always present when end value is present and always create a mock state
2. this let's you to run several chunks in parallel. the state is never written to (and possibly also not read)
3. the incremental loading will start when user calls the resource without passing end date. it is user responsibility to pass the correct initial value (which is the highest end date used by back loading). interestingly the incremental load also can happen in parallel

what do you think?

[steinitzu]

PR looks good but the whole concept raises some questions:

1. when having a concrete chunk to load we should IMO not modify the state at all. we may read the state to get ie. last_value or initial_value but I think it's better if we require initial value to be always present when end value is present and always create a mock state

2. this let's you to run several chunks in parallel. the state is never written to (and possibly also not read)

3. the incremental loading will start when user calls the resource without passing end date. it is user responsibility to pass the correct initial value (which is the highest end date used by back loading). interestingly the incremental load also can happen in parallel

what do you think?

Yeah, I agree.
The only benefit is you don't need to forward the last end_value to initial_value. Incremental just works after.
But that's not such a big deal compared to the weirdness you get from a bunch of tasks overwriting eachother's state.

And 👍 on requiring initial value. Should error also when initial value is higher, i.e. last_value_func(end_value, initial_value) == initial_value

[steinitzu]

@rudolfix updated like this in last commit. Using dummy state dict for this, and added a test to make sure the state isn't updated.

[rudolfix]

excellent! I've added a short paragraph on end_value to our docs.