Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some entities should be recomputed every time (or with some cadence) #27

Closed
jqmp opened this issue Oct 8, 2019 · 0 comments · Fixed by #95
Closed

Some entities should be recomputed every time (or with some cadence) #27

jqmp opened this issue Oct 8, 2019 · 0 comments · Fixed by #95
Labels
enhancement New feature or request

Comments

@jqmp
Copy link
Collaborator

jqmp commented Oct 8, 2019

Currently, an entity's value is only recomputed if its inputs change or its code changes. However, sometimes people want entities to be recomputed whenever some amount of time has passed. For example, we may want to download new data from a database each day. Right now the best workaround is like this:

builder.assign('date', datetime.today().strftime('%Y-%m-%d'))

@builder
@builder
def current_data(date):
    return download_data()

This is a bit hacky and doesn't work if the flow is long-lived, because date only gets updated when the flow is defined.

Note that the following does not work:

@builder
@bn.persist(False)
@builder
def current_data():
    return download_data()

@builder
def summary(current_data):
    return summarize(current_data)

This fails because if we call flow.get('summary') and it finds a cached value, it will return it because it doesn't know that current_data ought to be recomputed. We need to invalidate not just current_data, but all downstream entities as well.

A better approach would be a way to tell Bionic that (a) an entity should be recomputed every time, and (b) downstream entities should only be recomputed if this entity's output changes. Maybe something like:

@builder
@bn.volatile    # Always recompute
@bn.digestible  # Hash the output to determine whether to recompute children
def date():
    return datetime.today().strftime('%Y-%m-%d')

@builder
def current_data(date):
    return download_data()

I'm not sure about the names, or about needing two separate decorators (although I do think they would be useful independently).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant