Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

added lancedb contrib #527

Merged
merged 2 commits into from
Nov 10, 2023
Merged

added lancedb contrib #527

merged 2 commits into from
Nov 10, 2023

Conversation

zilto
Copy link
Collaborator

@zilto zilto commented Nov 10, 2023

The contrib allows to interact with lancedb by specifying a schema and facilitating read / writes.

For new dataflows:

Do you have the following?

  • Added a new folder for my dataflow under my username directory.
    • Added a README.md file under my dataflow directory that follows the standard headings and is filled out.
    • Added a init.py file under my dataflow directory that contains the Hamilton code.
    • Added a requirements.txt under my dataflow directory that contains the required packages outside of Hamilton.
    • Added tags.json under my dataflow directory to curate my dataflow.
    • Added valid_configs.jsonl under my dataflow directory to specify the valid configurations.
    • Added a dag.png that shows one possible configuration of my dataflow.

For existing dataflows -- what has changed?

How I tested this

Tested functions using the included run.ipynb notebook

Notes

Functions are annotated with @tag(side_effect="True") to indicate the interaction with the database.

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Dataflow documentation has been updated if adding/changing functionality.

@zilto zilto added the contrib Used for code related to contrib package label Nov 10, 2023
Copy link
Contributor

sweep-ai bot commented Nov 10, 2023

Apply Sweep Rules to your PR?

  • Apply: All new business logic should have corresponding unit tests.
  • Apply: Refactor large functions to be more modular.

@skrawcz skrawcz merged commit ea2e21d into main Nov 10, 2023
20 of 24 checks passed
@skrawcz skrawcz deleted the contrib/lancedb branch November 10, 2023 20:49


@tag(side_effect="True")
def push_data(table_ref: lancedb.db.LanceTable, data: Any) -> Dict:
Copy link
Collaborator

@skrawcz skrawcz Nov 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zilto oops -- merged too soon -- we should annotate data with dict, list[dict], pd.DataFrame, or a Iterator[pa.RecordBatch]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
contrib Used for code related to contrib package
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants