Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

key value store #4499

Merged
merged 30 commits into from Jun 1, 2021
Merged

key value store #4499

merged 30 commits into from Jun 1, 2021

Conversation

zangell44
Copy link
Collaborator

@zangell44 zangell44 commented May 8, 2021

Summary

Adds KV Store interfaces to Core.

Changes

This PR adds

  • KV Store backend functions
  • KV Store CLI
  • KV Store documentation

Importance

KV Store!

Checklist

This PR:

  • adds new tests (if appropriate)
  • adds a change file in the changes/ directory (if appropriate)
  • updates docstrings for any new functions or function arguments, including docs/outline.toml for API reference docs (if appropriate)

@zangell44 zangell44 requested a review from jcrist as a code owner May 8, 2021 17:13
@zangell44 zangell44 requested a review from zanieb May 8, 2021 17:13
@zangell44 zangell44 marked this pull request as draft May 8, 2021 17:14
src/prefect/__init__.py Outdated Show resolved Hide resolved
src/prefect/cli/key_value.py Outdated Show resolved Hide resolved
src/prefect/cli/key_value.py Outdated Show resolved Hide resolved
src/prefect/cli/key_value.py Outdated Show resolved Hide resolved
src/prefect/cli/key_value.py Outdated Show resolved Hide resolved
src/prefect/client/client.py Outdated Show resolved Hide resolved
src/prefect/client/client.py Outdated Show resolved Hide resolved
src/prefect/utilities/key_value.py Outdated Show resolved Hide resolved
@zangell44 zangell44 marked this pull request as ready for review May 21, 2021 01:43
@zangell44 zangell44 changed the title [DRAFT] key value store key value store May 21, 2021
@zangell44 zangell44 requested a review from zanieb May 21, 2021 14:41
zanieb
zanieb previously requested changes May 21, 2021
Copy link
Contributor

@zanieb zanieb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good!

docs/orchestration/concepts/kv_store.md Outdated Show resolved Hide resolved
src/prefect/backend/kv_store.py Show resolved Hide resolved
src/prefect/backend/kv_store.py Outdated Show resolved Hide resolved
src/prefect/cli/kv_store.py Outdated Show resolved Hide resolved
tests/backend/test_kv_store.py Outdated Show resolved Hide resolved
src/prefect/backend/__init__.py Outdated Show resolved Hide resolved
@zangell44 zangell44 requested review from zanieb and cicdw May 21, 2021 19:31
Copy link
Member

@zhen0 zhen0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some small suggestions for the docs. I also wonder if we could add an example of using KV in a flow - maybe in the KV store docs or an advanced section of the tutorial or in a blog post?

docs/orchestration/concepts/kv_store.md Outdated Show resolved Hide resolved
docs/orchestration/concepts/kv_store.md Outdated Show resolved Hide resolved
@zangell44
Copy link
Collaborator Author

Good call @zhen0! I added a simple example based on previous community feedback. Let me know what you think

Copy link
Contributor

@zanieb zanieb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some minor details

docs/orchestration/concepts/kv_store.md Outdated Show resolved Hide resolved
docs/orchestration/concepts/kv_store.md Outdated Show resolved Hide resolved
docs/orchestration/concepts/kv_store.md Outdated Show resolved Hide resolved
docs/orchestration/concepts/kv_store.md Outdated Show resolved Hide resolved
docs/orchestration/concepts/kv_store.md Outdated Show resolved Hide resolved
src/prefect/cli/kv_store.py Outdated Show resolved Hide resolved
src/prefect/cli/kv_store.py Outdated Show resolved Hide resolved
src/prefect/cli/kv_store.py Outdated Show resolved Hide resolved
src/prefect/cli/kv_store.py Outdated Show resolved Hide resolved
tests/backend/test_kv_store.py Outdated Show resolved Hide resolved
@zangell44 zangell44 requested a review from zanieb May 24, 2021 22:29
changes/pr4499.yaml Outdated Show resolved Hide resolved
zanieb
zanieb previously approved these changes May 24, 2021
zanieb
zanieb previously approved these changes May 24, 2021
@@ -4,7 +4,7 @@ Key Value Store is a managed metadata database for Prefect Cloud.

**Keys** are strings. **Values** are JSON blobs.

The number of key value pairs allowed is limited by license, starting with 50 pairs on the Free tier. Values are limited to 1 MB in size.
The number of key value pairs allowed is limited by license, starting with 10 pairs on the Free tier. Values are limited to 1 KB in size.
Copy link
Contributor

@zanieb zanieb May 28, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is wrong now. Sorry out of date

I think we should check sizes locally before sending to Cloud as well?

@zanieb zanieb merged commit de16481 into master Jun 1, 2021
@zanieb zanieb deleted the kv-store branch June 1, 2021 16:02
@zanieb zanieb mentioned this pull request Jun 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants