Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unify gather and get APIs #46

Open
jqmp opened this issue Oct 30, 2019 · 0 comments
Open

Unify gather and get APIs #46

jqmp opened this issue Oct 30, 2019 · 0 comments
Labels
enhancement New feature or request

Comments

@jqmp
Copy link
Collaborator

jqmp commented Oct 30, 2019

Entity values in Bionic can be accessed in two ways: from another entity definition, or directly from a flow with Flow.get. Currently these two APIs offer different ways to transform and aggregate the values.

When accessing entity values from the definition of another entity, there's a powerful @gather decorator that joins many values of many entities into a single dataframe. When using Flow.get, one can only aggregate the values of a single entity. However, get provides other options for accessing serialized values, which aren't available from within entity definitions.

These discrepancies make certain constructions impossible, and they also make the system harder to learn. In particular, new users often find it difficult to understand the @gather decorator -- I believe this is partly because there's no way to access gathered values directly from a flow, which makes it hard to experiment.

We should move towards a unified API where the same functionality is available in both circumstances. Perhaps something like these:

@builder
@bn.arg('gather_df', ['hyperparams', 'model_type'], also='performance', collection='frame')
def best_performance(gather_df):
    ...
@builder
@bn.arg('big_dataframe_filename', 'big_dataframe', mode='filename'):
def big_dataframe_filesize(big_dataframe_filename):
    return os.stat(big_dataframe_filename).st_size
flow.get(['hyperparams', 'model_type'], also='performance', collection='frame')
@jqmp jqmp added the enhancement New feature or request label Feb 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant