TypeAnnotation #759

kennyworkman · 2021-12-01T16:57:25Z

TL;DR

Use new flyte core object to saturate an Annotation field on flyte literal types

Type

Bug Fix
Feature
Plugin

Are all requirements met?

Code completed
Smoke tested
Unit tests added
[] Code documentation added
Any pending items have an associated Issue

Complete description

How did you fix the bug, make the feature etc. Link to any design docs etc

Tracking Issue

flyteorg/flyte#1855

Follow-up issue

NA

welcome · 2021-12-01T16:57:27Z

Thank you for opening this pull request! 🙌

These tips will help get your PR across the finish line:

Most of the repos have a PR template; if not, fill it out to the best of your knowledge.
Sign off your commits (Reference: DCO Guide).

eapolinario

Thanks for making this contribution! I left a few comments, but nothing too major. Overall this is shaping up to be really amazing.

flytekit/core/type_engine.py

kennyworkman · 2021-12-02T06:12:02Z

@eapolinario comments have been addressed

eapolinario · 2021-12-03T18:17:15Z

Will take another look after we merge flyteorg/flyteidl#232 and incorporate it to this PR.

wild-endeavor

Thank you for doing this! Made some comments... not sure that they all make sense. I haven't looked very deeply at all into this Annotated stuff. Didn't even know it existed until a couple months ago. Apologies if I'm off the mark.

flytekit/core/type_engine.py

flytekit/core/annotation.py

flytekit/core/type_engine.py

flytekit/models/annotation.py

tests/flytekit/unit/core/test_typing_annotation.py

flytekit/core/type_engine.py

eapolinario

This is looking good, I have just a few questions and a concern about the composition of annotated types, so I asked for more unit tests. Otherwise, this is shaping up to be really nice.

Don't worry too much about the "nit:" comments as the pre-commit hook runs black and other linters (so those stylistic comments should not be a concern).

flytekit/models/annotation.py

tests/flytekit/unit/core/test_type_engine.py

tests/flytekit/unit/core/test_typing_annotation.py

tests/flytekit/unit/core/test_type_engine.py

flytekit/core/type_engine.py

flytekit/models/annotation.py

tests/flytekit/unit/models/test_types.py

eapolinario

Left a few more comments, one comment especially is going to unblock unit tests. Also, once https://github.com/flyteorg/flytekit/pull/759/files#r766346591 is handled we should be close.

requirements.txt

flytekit/core/type_engine.py

flytekit/models/types.py

flytekit/core/annotation.py

eapolinario · 2021-12-14T22:30:48Z

@kennyworkman , can you merge master? Integration tests are passing in master again.

codecov · 2021-12-14T23:53:22Z

Codecov Report

Merging #759 (88bc4d0) into master (a3a9684) will increase coverage by 0.04%.
The diff coverage is 92.90%.

@@            Coverage Diff             @@
##           master     #759      +/-   ##
==========================================
+ Coverage   87.19%   87.24%   +0.04%     
==========================================
  Files         248      251       +3     
  Lines       22033    22181     +148     
  Branches     2457     2475      +18     
==========================================
+ Hits        19211    19351     +140     
- Misses       2396     2402       +6     
- Partials      426      428       +2

Impacted Files	Coverage Δ
flytekit/core/type_engine.py	`89.26% <84.84%> (+0.15%)`	⬆️
flytekit/models/types.py	`97.25% <87.50%> (-0.45%)`	⬇️
flytekit/models/annotation.py	`89.47% <89.47%> (ø)`
tests/flytekit/unit/core/test_typing_annotation.py	`92.30% <92.30%> (ø)`
flytekit/core/annotation.py	`100.00% <100.00%> (ø)`
tests/flytekit/unit/core/test_type_engine.py	`99.66% <100.00%> (+0.02%)`	⬆️
tests/flytekit/unit/models/test_types.py	`100.00% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a3a9684...88bc4d0. Read the comment docs.

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

eapolinario

This is amazing work, @kennyworkman! Thanks for pushing through a lengthy review process, I hope we can make this smoother in the future.

welcome · 2022-02-08T19:36:49Z

Congrats on merging your first pull request! 🎉

* feat: support for annotated simple + list Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: addition of annotation att to Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: core obj Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: proto model Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: testing suite Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: more stable typing introspection Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: strip legacy Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: explicitly allow only one annotation Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: direct type transformer tests Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: there and back test Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: typing_extensions for get_origin Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: more semantic list generic unwrap Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: tmp requirements file with custom idl Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: nits Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: semantic error for unsupported complex literals Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: but Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: more tests ;) Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: imports Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: complex annotations Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: temp requirements files for unit tests Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: lint bug Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: tmp setup.py Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: use typing_extensions Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: typing_extensions for annotated Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: typing_ext Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: plugin tmp requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: bump requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: doc requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: whitespace Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: bump flytekit Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: numpy version Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: lint Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: pandas version Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: bump requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: test import Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: flake8 lint Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: merge Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: lint Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: papermill req Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: req Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

@task

* Add support union type Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed test Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed test Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Update Union to use tagged Unions Signed-off-by: maximsmol <maximsmol@gmail.com> * Update to use string tags (part 1) Signed-off-by: maximsmol <maximsmol@gmail.com> * Working implementation, update tests Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixes, more tests Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix incorrect unwrapped literal-union matching, update test Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix duplicate tag handling, add tests for collections containing unions Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix type hint test Signed-off-by: maximsmol <maximsmol@gmail.com> * Add implicit wrapping union type tests Signed-off-by: maximsmol <maximsmol@gmail.com> * Add union ambiguity tests Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixup tests, make TypeTransformerFailed compatible with all old exception types Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixup models + add tests Signed-off-by: maximsmol <maximsmol@gmail.com> * Implement changed design Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix PR comments Signed-off-by: maximsmol <maximsmol@gmail.com> * Remote entrypoint serialize (#733) Signed-off-by: Emirhan Karagül <emirhan350z@gmail.com> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix lint error in remote.py (#755) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Support enum in dataclass (#753) * Add support enum in dataclass Signed-off-by: Kevin Su <pingsutw@apache.org> * Update test Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix subworkflow and launch plan FlyteRemote behavior (#751) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Extras: Shell task (#747) Signed-off-by: maximsmol <maximsmol@gmail.com> * Add support FlyteSchema in dataclass (#722) * schema in dataclass Signed-off-by: Kevin Su <pingsutw@apache.org> * Added tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * updated Signed-off-by: Kevin Su <pingsutw@apache.org> * updated Signed-off-by: Kevin Su <pingsutw@apache.org> * updated Signed-off-by: Kevin Su <pingsutw@apache.org> * Remove workflow_execution.py (#758) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Get raw input/output from remote execution (#675) * [wip] for feast demo Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * clean up a bit Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add a test and move where constructor is called Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * remove unneeded import Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add a part of a test Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Added tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * typo Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix mypy errors in flytekit/types (#757) Signed-off-by: Lisa <aeioulisa@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> * Remote client failed to fetch FlytePickle object (#764) * Fetch pickle value from flytekit remote Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Remove default value Signed-off-by: Kevin Su <pingsutw@apache.org> * Add support FlyteFile in dataclass (#725) * Add support Flyte File and directory in dataclass Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * add task_resolver arg to @task decorator (#765) Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Copy metadata into map task from underlying (#766) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Support for delayed annotations (#760) Signed-off-by: Stefan Nelson-Lindall <stef@stripe.com> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Complex dataclass unit tests (#773) Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * remote_source lost on serialization of @dataclass_json with FlyteFile (#774) * remote_source lost on serialization of @dataclass_json with FlyteFile Signed-off-by: Kevin Su <pingsutw@apache.org> * updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> * updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Single-task execution FlyteRemote sync (#778) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Logging updates (#775) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: doc-requirements.txt to reduce vulnerabilities (#779) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-LXML-2316995 Signed-off-by: maximsmol <maximsmol@gmail.com> * Add cache_serialize parameter to tasks (#673) * added cache_serialize parameter for basic tasks Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed typo Signed-off-by: Daniel Rammer <daniel@union.ai> * updated flyteidl version Signed-off-by: Daniel Rammer <daniel@union.ai> * changed flyteidl requirements everywhere Signed-off-by: Daniel Rammer <daniel@union.ai> * remove flyteidl version requires in setup.py so we can use develop Signed-off-by: Daniel Rammer <daniel@union.ai> * removed flyteidl git repos from a variety of requirements packages Signed-off-by: Daniel Rammer <daniel@union.ai> * updated variable discovery_serializable to cache_serializable Signed-off-by: Daniel Rammer <daniel@union.ai> * updated requirements Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed TaskMetadata _cache_serializable variable name Signed-off-by: Daniel Rammer <daniel@union.ai> * propgating cache_serialize parameter through to tasks Signed-off-by: Daniel Rammer <daniel@union.ai> * added cache_serializable to SdkRawContainerTask Signed-off-by: Daniel Rammer <daniel@union.ai> * fixing cache_serializable variable propogation issues Signed-off-by: Daniel Rammer <daniel@union.ai> * added documentation Signed-off-by: Daniel Rammer <daniel@union.ai> * added unit tests for cache_serialize metadata Signed-off-by: Daniel Rammer <daniel@union.ai> * linter added spaces in unit tests Signed-off-by: Daniel Rammer <daniel@union.ai> Signed-off-by: maximsmol <maximsmol@gmail.com> * When using the `task` and `workflow` decorator, correctly wrap the fu… (#780) * When using the `task` and `workflow` decorator, correctly wrap the function This enables tooling such as docstring search tools to unwrap the object and show the correct docstring. Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Remove blackshark copyright header Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Fix broken great expectations test Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Add test for stacked decorators Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> Co-authored-by: Bernhard Stadlbauer <bstadlbauer@gmx.net> Co-authored-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add option to flyte-cli for specifying root certificate (#783) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add validation check to cacert switch (#787) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Remove pytz constraint (#786) * Remove pytz constraint from setup.py Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Put pytz back Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * make requirements.txt Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Lint: remove f-string misuse (#788) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Pyarrow greater than 4.0.0 (#790) * pyarrow>=4.0.0 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate requirements. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Support python 3.10 (#791) * [wip] Support python 3.10 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add Dockerfile.py310 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Stringify python version Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Skip flytekit-modin plugin tests on 3.10 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add 3.9 and 3.10 to list of supported version in plugins Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Comment why flytekit-modin is not running on 3.10 and disable fail-fast in plugin tests Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * add `with_overrides` to map task (#794) * add with_overrides Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * remove Resources Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * bump docsearch version (#805) Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * update docs for new navbar theme (#806) Signed-off-by: maximsmol <maximsmol@gmail.com> * fix requirment.txt github issue (#810) Signed-off-by: Yuvraj <code@evalsocket.dev> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add sphinx panels (#815) Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Schema overhaul (#785) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Parent workflow serialization fails when calling a launch plan with fixed inputs (#814) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix sagemaker plugin (#817) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Remove legacy API (#807) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Add Bigquery plugin (#789) * Add bigquery plugin Signed-off-by: Kevin Su <pingsutw@apache.org> * Update dependency Signed-off-by: Kevin Su <pingsutw@apache.org> * update get_custom Signed-off-by: Kevin Su <pingsutw@apache.org> * Add structured dataset Signed-off-by: Kevin Su <pingsutw@apache.org> * Add structured dataset Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated comment Signed-off-by: Kevin Su <pingsutw@apache.org> * Add BQ in GA Signed-off-by: Kevin Su <pingsutw@apache.org> * alphabetical order Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed flytekit-papermill ImportError (#818) Signed-off-by: Kevin Su <pingsutw@apache.org> * Add support for string-format-like sytax for shell task (#792) * POC: Add support for f-string like sytax for shell task This commit is a proof of concept adding f-string like syntax for shell_tasks. This supports using nested types for script inputs, such as data classes. This change was motivated by the desire to combine shell_tasks that have multiple inputs with map_tasks which only support tasks with a single input. This commit is only a starting point, since it makes some changes to the shell_task API (adds a template_style field), and modifies some of the default behavior for ease of implementation (e.g. throwing an error when there are unused input arguments). Signed-off-by: Zach Palchick <palchicz@zymergen.com> * Drop support for old/regex style for doing string interpolation Signed-off-by: Zach Palchick <palchicz@zymergen.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Expose configured RawOutputPrefix during execution (#813) * Expose configured RawOutputPrefix during execution Signed-off-by: Kevin Su <pingsutw@apache.org> * Remove sdk_runnable.py and spark_task.py Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add SecretsManager back to old import location (#820) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add some tests (#819) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed flaky spark test (#821) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: plugins/flytekit-greatexpectations/requirements.txt to reduce vulnerabilities (#823) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: plugins/flytekit-papermill/requirements.txt to reduce vulnerabilities (#825) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: requirements-spark2.txt to reduce vulnerabilities (#826) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: requirements.txt to reduce vulnerabilities (#824) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <maximsmol@gmail.com> * Intratask checkpointing (#771) * wip - intratask checkpointing Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * sync checkpointer with tests Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Checkpinter in entrypoint Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * checkpoint in progress Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * wip Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Intratask checkpointer Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Checkpoint updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Intra-task checkpointing Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Test and entrypoint updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * lint fixed Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * test fixes Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fmt Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated entrypoint Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * update Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * print Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * SyncCheckpointer working Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * update Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed import problems Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed test Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed imports Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed lints and errors Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * lint fix Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * addressed comments Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Support reading subset column (#822) * Support StructuredDatasetDecoder read subset column Signed-off-by: Kevin Su <pingsutw@apache.org> * Added tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed typo Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> * [pr into #822] (#827) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> * [pr into #822] Final update to structured dataset column subsetting (#828) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Fix spark regression (#830) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Update argument setting for in fast registered, dynamically generated, pod tasks (#835) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * `ctx` Context can be used within shell tasks - to access context vars and secrets (#832) * Adding context to a substitutable parameter in shell task Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Support for secrets in context Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * addressed comments Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Expose Checkpoint as a top-level interface in flytekit (#839) Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Parse duration field from flyteidl to `flytekit.models.execution.ExecutionClosure` (#829) * Parse duration field from flyteidl to `flytekit.models.execution.ExecutionClosure` Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Add test for execution closure Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Add tests to Flyte remote Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Split execution test into with output and with error Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> Co-authored-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> Signed-off-by: maximsmol <maximsmol@gmail.com> * Gate new Structured Dataset feature & remove old config objects (#831) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Fixing out of order for conditional outputs (#843) Signed-off-by: maximsmol <maximsmol@gmail.com> * Set default values to map task template (#841) * Set sane defaults in map task templates Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove unused method Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Put ArrayJob.from_dict back Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Define parallelism=0 as unbounded Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove special case to handle 0 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Regenerate requirements files - dependencies dropping support for python 3.7 (#838) * Regenerate requirements files Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Put restrictions on numpy and pandas versions Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Use --use-deprecated=legacy-resolver Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * use pip==22.0.3 everywhere Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove --use-deprecated=legacy-resolver Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Relax click Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerated plugins requirements Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * TypeAnnotation (#759) * feat: support for annotated simple + list Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: addition of annotation att to Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: core obj Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: proto model Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: testing suite Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: more stable typing introspection Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: strip legacy Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: explicitly allow only one annotation Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: direct type transformer tests Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: there and back test Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: typing_extensions for get_origin Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: more semantic list generic unwrap Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: tmp requirements file with custom idl Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: nits Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: semantic error for unsupported complex literals Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: but Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: more tests ;) Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: imports Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: complex annotations Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: temp requirements files for unit tests Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: lint bug Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: tmp setup.py Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: use typing_extensions Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: typing_extensions for annotated Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: typing_ext Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: plugin tmp requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: bump requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: doc requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: whitespace Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: bump flytekit Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: numpy version Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: lint Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: pandas version Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: bump requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: test import Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: flake8 lint Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: merge Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: lint Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: papermill req Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: req Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * Remove singleton from structured dataset transformer engine (#848) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> * Assign input and output to FlyteWorkflowExecution (#842) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add reference entities to FlyteTask and FlyteLaunchPlan (#850) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix fast registration error (#851) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add support for local execute in pod task (#852) Signed-off-by: ggydush-fn <greg.gydush@freenome.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add anonymous retry (#854) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: formatting, linting, typing_extensions * fix: do not use SDK types Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: update test comment Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: also check literal type castability when tags match Signed-off-by: maximsmol <maximsmol@gmail.com> * Point flyteidl to maxim's fork in CI and requirements files Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * [Core feature] Add Raw AWS Batch Task (#782) * Init plugin Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * address comment Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed typo Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated AWS config * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * Added comment Signed-off-by: Kevin Su <pingsutw@apache.org> * Update config Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * use pyflyte execute Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Added comment Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add structured dataset encoder/decoder in fsspec plugin (#849) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Delete unnecessary auth configuration (#858) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed format alias in Flytekit docs (#844) * Fixed format alias Fixed docs for file format alias that weren't rendered properly. A warning popped up stating 'ignore' is deprecated, and to use 'ignore_paths' instead. Signed-off-by: SmritiSatyanV <smriti@union.ai> * Bump idl (#862) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Updated authoring.rst (#863) * Updated authoring.rst Added directive Rephrased sentence * Fixed build error Signed-off-by: SmritiSatyanV <smriti@union.ai> * test-build-1 Signed-off-by: SmritiSatyanV <smriti@union.ai> Signed-off-by: maximsmol <maximsmol@gmail.com> * Updated authoring.rst (#866) Added the directive correctly Signed-off-by: SmritiSatyanV <smriti@union.ai> Signed-off-by: maximsmol <maximsmol@gmail.com> * Change docs for HTMLPage type to say HTMLPage instead of PNGImage (#868) Signed-off-by: maximsmol <maximsmol@gmail.com> * Revisit StructuredDatasetDecoder interface (#865) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Remove legacy mentions in contributing guide (#870) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add GCS protocol in the structured dataset (#869) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Make fetched entities callable within workflows (#867) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Caching of offloaded objects (#762) * Remove flyteidl from install_requires Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Expose hash in Literal Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Set hash in TypeEngine Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Modify cache key calculation to take hash into account Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Opt-in PandasDataFrameTransformer Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add unit tests Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Iterate using a flyteidl branch Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Move _hash_overridable to StructureDatasetTransformerEngine Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Move HashMethod to flytekit.core.hash Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Fix `unit_test` make target Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Split `unit_test` make target in two lines Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add assert to structured dataset compatibility test Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove TODO Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate plugins requirements files pointing to the right version of flyteidl. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Set hash as a property of the literal Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Install plugins requirements in CI. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add hash.setter Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Install flyteidl directly Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revert "Regenerate plugins requirements files pointing to the right version of flyteidl." This reverts commit c2dbb54. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * wip - Add support for univariate lists Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add support for lists of annotated objects Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revamp generation of cache key (to cover case of literals collections and maps) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Leave TODO for warning Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revert "Add support for lists of annotated objects" This reverts commit 4b5f608. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revert "wip - Add support for univariate lists" This reverts commit adaa448. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove docstring Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add flyteidl>=0.23.0 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove mentions to branch flyteidl@add-hash-to-literal Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Bump flyteidl in plugins requirements Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate plugins requirements again Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Restore papermill/requirements.txt Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Point flytekitplugins-spark to the offloaded-objects-caching branch in papermill tests Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Set flyteidl>=0.23.0 in papermill dev-requirements Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * FlyteRemote fetch of conditional nodes (#772) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Removed root logger (#871) * Removed root logger Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated logger Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated log level Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated logger Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated logger Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated logger Signed-off-by: Kevin Su <pingsutw@apache.org> * lint fixed Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix flytekit_compatibility/test_schema_types.py test Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * make lint Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * fix: annotated type conversion error Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: _are_types_castable based on tests Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: test failing if using random order Signed-off-by: maximsmol <maximsmol@gmail.com> * Merge branch 'master' into maximsmol/union_type Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: merge issue Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: requirements Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: schema transformer error Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: test Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: merge issue Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: union + annotated behavior Signed-off-by: maximsmol <maximsmol@gmail.com> * Regenerate requirements Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Bump requirements in plugins Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Handle nested Annotated Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Leave TODO re: strucutured dataset type castability Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove mention to flyteidl@union_type in doc-requirements Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Linting Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Use tempfile.mkdtemp to create a temporary directory for local data persistence. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revert "Use tempfile.mkdtemp to create a temporary directory for local data persistence." This reverts commit 861b157. Signed-off-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local> * Force temporary file to not be deleted in test Signed-off-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local> * Regenerate papermill dev-requirements Signed-off-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local> * Remove duplicate code Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Put a lower bound on the pip version installed in CI Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Emirhan Karagül <emirhan350z@gmail.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Kevin Su <pingsutw@gmail.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Ketan Umare <16888709+kumare3@users.noreply.github.com> Co-authored-by: Lisa <30621230+aeioulisa@users.noreply.github.com> Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com> Co-authored-by: Stef Nelson-Lindall <bethebunny@gmail.com> Co-authored-by: Snyk bot <github+bot@snyk.io> Co-authored-by: Dan Rammer <hamersaw@protonmail.com> Co-authored-by: bstadlbauer <11799671+bstadlbauer@users.noreply.github.com> Co-authored-by: Bernhard Stadlbauer <bstadlbauer@gmx.net> Co-authored-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> Co-authored-by: Samhita Alla <aallasamhita@gmail.com> Co-authored-by: Yuvraj <code@evalsocket.dev> Co-authored-by: Zach Palchick <palchicz@users.noreply.github.com> Co-authored-by: Snyk bot <snyk-bot@snyk.io> Co-authored-by: Kenny Workman <31255434+kennyworkman@users.noreply.github.com> Co-authored-by: ggydush-fn <69013027+ggydush-fn@users.noreply.github.com> Co-authored-by: SmritiSatyanV <94349093+SmritiSatyanV@users.noreply.github.com> Co-authored-by: Matthew Griffin <1matthewgriffin@gmail.com> Co-authored-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local>

@task

* Add support union type Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed test Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed test Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Update Union to use tagged Unions Signed-off-by: maximsmol <maximsmol@gmail.com> * Update to use string tags (part 1) Signed-off-by: maximsmol <maximsmol@gmail.com> * Working implementation, update tests Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixes, more tests Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix incorrect unwrapped literal-union matching, update test Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix duplicate tag handling, add tests for collections containing unions Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix type hint test Signed-off-by: maximsmol <maximsmol@gmail.com> * Add implicit wrapping union type tests Signed-off-by: maximsmol <maximsmol@gmail.com> * Add union ambiguity tests Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixup tests, make TypeTransformerFailed compatible with all old exception types Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixup models + add tests Signed-off-by: maximsmol <maximsmol@gmail.com> * Implement changed design Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix PR comments Signed-off-by: maximsmol <maximsmol@gmail.com> * Remote entrypoint serialize (flyteorg#733) Signed-off-by: Emirhan Karagül <emirhan350z@gmail.com> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix lint error in remote.py (flyteorg#755) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Support enum in dataclass (flyteorg#753) * Add support enum in dataclass Signed-off-by: Kevin Su <pingsutw@apache.org> * Update test Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix subworkflow and launch plan FlyteRemote behavior (flyteorg#751) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Extras: Shell task (flyteorg#747) Signed-off-by: maximsmol <maximsmol@gmail.com> * Add support FlyteSchema in dataclass (flyteorg#722) * schema in dataclass Signed-off-by: Kevin Su <pingsutw@apache.org> * Added tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * updated Signed-off-by: Kevin Su <pingsutw@apache.org> * updated Signed-off-by: Kevin Su <pingsutw@apache.org> * updated Signed-off-by: Kevin Su <pingsutw@apache.org> * Remove workflow_execution.py (flyteorg#758) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Get raw input/output from remote execution (flyteorg#675) * [wip] for feast demo Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * clean up a bit Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add a test and move where constructor is called Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * remove unneeded import Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * add a part of a test Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Added tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * typo Signed-off-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix mypy errors in flytekit/types (flyteorg#757) Signed-off-by: Lisa <aeioulisa@gmail.com> Signed-off-by: Kevin Su <pingsutw@apache.org> * Remote client failed to fetch FlytePickle object (flyteorg#764) * Fetch pickle value from flytekit remote Signed-off-by: Kevin Su <pingsutw@apache.org> * Fix tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Remove default value Signed-off-by: Kevin Su <pingsutw@apache.org> * Add support FlyteFile in dataclass (flyteorg#725) * Add support Flyte File and directory in dataclass Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * add task_resolver arg to @task decorator (flyteorg#765) Signed-off-by: Niels Bantilan <niels.bantilan@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Copy metadata into map task from underlying (flyteorg#766) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Support for delayed annotations (flyteorg#760) Signed-off-by: Stefan Nelson-Lindall <stef@stripe.com> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Complex dataclass unit tests (flyteorg#773) Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * remote_source lost on serialization of @dataclass_json with FlyteFile (flyteorg#774) * remote_source lost on serialization of @dataclass_json with FlyteFile Signed-off-by: Kevin Su <pingsutw@apache.org> * updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> * updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Single-task execution FlyteRemote sync (flyteorg#778) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Logging updates (flyteorg#775) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: doc-requirements.txt to reduce vulnerabilities (flyteorg#779) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-LXML-2316995 Signed-off-by: maximsmol <maximsmol@gmail.com> * Add cache_serialize parameter to tasks (flyteorg#673) * added cache_serialize parameter for basic tasks Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed typo Signed-off-by: Daniel Rammer <daniel@union.ai> * updated flyteidl version Signed-off-by: Daniel Rammer <daniel@union.ai> * changed flyteidl requirements everywhere Signed-off-by: Daniel Rammer <daniel@union.ai> * remove flyteidl version requires in setup.py so we can use develop Signed-off-by: Daniel Rammer <daniel@union.ai> * removed flyteidl git repos from a variety of requirements packages Signed-off-by: Daniel Rammer <daniel@union.ai> * updated variable discovery_serializable to cache_serializable Signed-off-by: Daniel Rammer <daniel@union.ai> * updated requirements Signed-off-by: Daniel Rammer <daniel@union.ai> * fixed TaskMetadata _cache_serializable variable name Signed-off-by: Daniel Rammer <daniel@union.ai> * propgating cache_serialize parameter through to tasks Signed-off-by: Daniel Rammer <daniel@union.ai> * added cache_serializable to SdkRawContainerTask Signed-off-by: Daniel Rammer <daniel@union.ai> * fixing cache_serializable variable propogation issues Signed-off-by: Daniel Rammer <daniel@union.ai> * added documentation Signed-off-by: Daniel Rammer <daniel@union.ai> * added unit tests for cache_serialize metadata Signed-off-by: Daniel Rammer <daniel@union.ai> * linter added spaces in unit tests Signed-off-by: Daniel Rammer <daniel@union.ai> Signed-off-by: maximsmol <maximsmol@gmail.com> * When using the `task` and `workflow` decorator, correctly wrap the fu… (flyteorg#780) * When using the `task` and `workflow` decorator, correctly wrap the function This enables tooling such as docstring search tools to unwrap the object and show the correct docstring. Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Remove blackshark copyright header Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Fix broken great expectations test Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Add test for stacked decorators Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> Co-authored-by: Bernhard Stadlbauer <bstadlbauer@gmx.net> Co-authored-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add option to flyte-cli for specifying root certificate (flyteorg#783) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add validation check to cacert switch (flyteorg#787) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Remove pytz constraint (flyteorg#786) * Remove pytz constraint from setup.py Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Put pytz back Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * make requirements.txt Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Lint: remove f-string misuse (flyteorg#788) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Pyarrow greater than 4.0.0 (flyteorg#790) * pyarrow>=4.0.0 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate requirements. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Support python 3.10 (flyteorg#791) * [wip] Support python 3.10 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add Dockerfile.py310 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Stringify python version Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Skip flytekit-modin plugin tests on 3.10 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add 3.9 and 3.10 to list of supported version in plugins Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Comment why flytekit-modin is not running on 3.10 and disable fail-fast in plugin tests Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * add `with_overrides` to map task (flyteorg#794) * add with_overrides Signed-off-by: Samhita Alla <aallasamhita@gmail.com> * remove Resources Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * bump docsearch version (flyteorg#805) Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * update docs for new navbar theme (flyteorg#806) Signed-off-by: maximsmol <maximsmol@gmail.com> * fix requirment.txt github issue (flyteorg#810) Signed-off-by: Yuvraj <code@evalsocket.dev> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add sphinx panels (flyteorg#815) Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Schema overhaul (flyteorg#785) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Parent workflow serialization fails when calling a launch plan with fixed inputs (flyteorg#814) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix sagemaker plugin (flyteorg#817) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Remove legacy API (flyteorg#807) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Add Bigquery plugin (flyteorg#789) * Add bigquery plugin Signed-off-by: Kevin Su <pingsutw@apache.org> * Update dependency Signed-off-by: Kevin Su <pingsutw@apache.org> * update get_custom Signed-off-by: Kevin Su <pingsutw@apache.org> * Add structured dataset Signed-off-by: Kevin Su <pingsutw@apache.org> * Add structured dataset Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated comment Signed-off-by: Kevin Su <pingsutw@apache.org> * Add BQ in GA Signed-off-by: Kevin Su <pingsutw@apache.org> * alphabetical order Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed flytekit-papermill ImportError (flyteorg#818) Signed-off-by: Kevin Su <pingsutw@apache.org> * Add support for string-format-like sytax for shell task (flyteorg#792) * POC: Add support for f-string like sytax for shell task This commit is a proof of concept adding f-string like syntax for shell_tasks. This supports using nested types for script inputs, such as data classes. This change was motivated by the desire to combine shell_tasks that have multiple inputs with map_tasks which only support tasks with a single input. This commit is only a starting point, since it makes some changes to the shell_task API (adds a template_style field), and modifies some of the default behavior for ease of implementation (e.g. throwing an error when there are unused input arguments). Signed-off-by: Zach Palchick <palchicz@zymergen.com> * Drop support for old/regex style for doing string interpolation Signed-off-by: Zach Palchick <palchicz@zymergen.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Expose configured RawOutputPrefix during execution (flyteorg#813) * Expose configured RawOutputPrefix during execution Signed-off-by: Kevin Su <pingsutw@apache.org> * Remove sdk_runnable.py and spark_task.py Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add SecretsManager back to old import location (flyteorg#820) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add some tests (flyteorg#819) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed flaky spark test (flyteorg#821) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: plugins/flytekit-greatexpectations/requirements.txt to reduce vulnerabilities (flyteorg#823) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: plugins/flytekit-papermill/requirements.txt to reduce vulnerabilities (flyteorg#825) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: requirements-spark2.txt to reduce vulnerabilities (flyteorg#826) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: requirements.txt to reduce vulnerabilities (flyteorg#824) The following vulnerabilities are fixed by pinning transitive dependencies: - https://snyk.io/vuln/SNYK-PYTHON-IPYTHON-2348630 Signed-off-by: maximsmol <maximsmol@gmail.com> * Intratask checkpointing (flyteorg#771) * wip - intratask checkpointing Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * sync checkpointer with tests Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Checkpinter in entrypoint Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * checkpoint in progress Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * wip Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Intratask checkpointer Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Checkpoint updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Intra-task checkpointing Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Test and entrypoint updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * lint fixed Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * test fixes Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fmt Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated entrypoint Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * update Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * print Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * SyncCheckpointer working Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * updated Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * update Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed import problems Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed test Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed imports Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * fixed lints and errors Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * lint fix Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * addressed comments Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Support reading subset column (flyteorg#822) * Support StructuredDatasetDecoder read subset column Signed-off-by: Kevin Su <pingsutw@apache.org> * Added tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed typo Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated tests Signed-off-by: Kevin Su <pingsutw@apache.org> * [pr into flyteorg#822] (flyteorg#827) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> * [pr into flyteorg#822] Final update to structured dataset column subsetting (flyteorg#828) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Fix spark regression (flyteorg#830) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Update argument setting for in fast registered, dynamically generated, pod tasks (flyteorg#835) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * `ctx` Context can be used within shell tasks - to access context vars and secrets (flyteorg#832) * Adding context to a substitutable parameter in shell task Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * Support for secrets in context Signed-off-by: Ketan Umare <ketan.umare@gmail.com> * addressed comments Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Expose Checkpoint as a top-level interface in flytekit (flyteorg#839) Signed-off-by: Ketan Umare <ketan.umare@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Parse duration field from flyteidl to `flytekit.models.execution.ExecutionClosure` (flyteorg#829) * Parse duration field from flyteidl to `flytekit.models.execution.ExecutionClosure` Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Add test for execution closure Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Add tests to Flyte remote Signed-off-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> * Split execution test into with output and with error Signed-off-by: Bernhard Stadlbauer <b.stadlbauer@gmx.net> Co-authored-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> Signed-off-by: maximsmol <maximsmol@gmail.com> * Gate new Structured Dataset feature & remove old config objects (flyteorg#831) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Fixing out of order for conditional outputs (flyteorg#843) Signed-off-by: maximsmol <maximsmol@gmail.com> * Set default values to map task template (flyteorg#841) * Set sane defaults in map task templates Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove unused method Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Put ArrayJob.from_dict back Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Define parallelism=0 as unbounded Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove special case to handle 0 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Regenerate requirements files - dependencies dropping support for python 3.7 (flyteorg#838) * Regenerate requirements files Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Put restrictions on numpy and pandas versions Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Use --use-deprecated=legacy-resolver Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * use pip==22.0.3 everywhere Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove --use-deprecated=legacy-resolver Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Relax click Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerated plugins requirements Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * TypeAnnotation (flyteorg#759) * feat: support for annotated simple + list Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: addition of annotation att to Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: core obj Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: proto model Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: testing suite Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: more stable typing introspection Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: strip legacy Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: explicitly allow only one annotation Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: direct type transformer tests Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: there and back test Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: typing_extensions for get_origin Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: more semantic list generic unwrap Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: tmp requirements file with custom idl Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: nits Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: semantic error for unsupported complex literals Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: but Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * feat: more tests ;) Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: imports Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: complex annotations Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: temp requirements files for unit tests Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: lint bug Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: tmp setup.py Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: use typing_extensions Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: typing_extensions for annotated Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: typing_ext Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: plugin tmp requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: bump requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: doc requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: whitespace Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: bump flytekit Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: numpy version Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: lint Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: pandas version Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: bump requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: test import Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: flake8 lint Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: merge Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: requirements Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: lint Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: papermill req Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * fix: req Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net> * Remove singleton from structured dataset transformer engine (flyteorg#848) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Kevin Su <pingsutw@apache.org> * Assign input and output to FlyteWorkflowExecution (flyteorg#842) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add reference entities to FlyteTask and FlyteLaunchPlan (flyteorg#850) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix fast registration error (flyteorg#851) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: Samhita Alla <aallasamhita@gmail.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add support for local execute in pod task (flyteorg#852) Signed-off-by: ggydush-fn <greg.gydush@freenome.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add anonymous retry (flyteorg#854) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: formatting, linting, typing_extensions * fix: do not use SDK types Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: update test comment Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: also check literal type castability when tags match Signed-off-by: maximsmol <maximsmol@gmail.com> * Point flyteidl to maxim's fork in CI and requirements files Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * [Core feature] Add Raw AWS Batch Task (flyteorg#782) * Init plugin Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * address comment Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed typo Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated AWS config * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * Added comment Signed-off-by: Kevin Su <pingsutw@apache.org> * Update config Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * use pyflyte execute Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed tests Signed-off-by: Kevin Su <pingsutw@apache.org> * Added comment Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add structured dataset encoder/decoder in fsspec plugin (flyteorg#849) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Delete unnecessary auth configuration (flyteorg#858) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fixed format alias in Flytekit docs (flyteorg#844) * Fixed format alias Fixed docs for file format alias that weren't rendered properly. A warning popped up stating 'ignore' is deprecated, and to use 'ignore_paths' instead. Signed-off-by: SmritiSatyanV <smriti@union.ai> * Bump idl (flyteorg#862) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> * Updated authoring.rst (flyteorg#863) * Updated authoring.rst Added directive Rephrased sentence * Fixed build error Signed-off-by: SmritiSatyanV <smriti@union.ai> * test-build-1 Signed-off-by: SmritiSatyanV <smriti@union.ai> Signed-off-by: maximsmol <maximsmol@gmail.com> * Updated authoring.rst (flyteorg#866) Added the directive correctly Signed-off-by: SmritiSatyanV <smriti@union.ai> Signed-off-by: maximsmol <maximsmol@gmail.com> * Change docs for HTMLPage type to say HTMLPage instead of PNGImage (flyteorg#868) Signed-off-by: maximsmol <maximsmol@gmail.com> * Revisit StructuredDatasetDecoder interface (flyteorg#865) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Remove legacy mentions in contributing guide (flyteorg#870) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Add GCS protocol in the structured dataset (flyteorg#869) Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Make fetched entities callable within workflows (flyteorg#867) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Caching of offloaded objects (flyteorg#762) * Remove flyteidl from install_requires Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Expose hash in Literal Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Set hash in TypeEngine Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Modify cache key calculation to take hash into account Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Opt-in PandasDataFrameTransformer Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add unit tests Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Iterate using a flyteidl branch Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate requirements files Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Move _hash_overridable to StructureDatasetTransformerEngine Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Move HashMethod to flytekit.core.hash Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Fix `unit_test` make target Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Split `unit_test` make target in two lines Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add assert to structured dataset compatibility test Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove TODO Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate plugins requirements files pointing to the right version of flyteidl. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Set hash as a property of the literal Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Install plugins requirements in CI. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add hash.setter Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Install flyteidl directly Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revert "Regenerate plugins requirements files pointing to the right version of flyteidl." This reverts commit c2dbb54. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * wip - Add support for univariate lists Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add support for lists of annotated objects Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revamp generation of cache key (to cover case of literals collections and maps) Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Leave TODO for warning Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revert "Add support for lists of annotated objects" This reverts commit 4b5f608. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revert "wip - Add support for univariate lists" This reverts commit adaa448. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove docstring Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Add flyteidl>=0.23.0 Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove mentions to branch flyteidl@add-hash-to-literal Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Bump flyteidl in plugins requirements Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Regenerate plugins requirements again Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Restore papermill/requirements.txt Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Point flytekitplugins-spark to the offloaded-objects-caching branch in papermill tests Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Set flyteidl>=0.23.0 in papermill dev-requirements Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * FlyteRemote fetch of conditional nodes (flyteorg#772) Signed-off-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * Removed root logger (flyteorg#871) * Removed root logger Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated logger Signed-off-by: Kevin Su <pingsutw@apache.org> * Fixed lint Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated log level Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated logger Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated logger Signed-off-by: Kevin Su <pingsutw@apache.org> * Updated logger Signed-off-by: Kevin Su <pingsutw@apache.org> * lint fixed Signed-off-by: Kevin Su <pingsutw@apache.org> Signed-off-by: maximsmol <maximsmol@gmail.com> * Fix flytekit_compatibility/test_schema_types.py test Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Signed-off-by: maximsmol <maximsmol@gmail.com> * make lint Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * fix: annotated type conversion error Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: _are_types_castable based on tests Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: test failing if using random order Signed-off-by: maximsmol <maximsmol@gmail.com> * Merge branch 'master' into maximsmol/union_type Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: merge issue Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: requirements Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: schema transformer error Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: test Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: merge issue Signed-off-by: maximsmol <maximsmol@gmail.com> * fix: union + annotated behavior Signed-off-by: maximsmol <maximsmol@gmail.com> * Regenerate requirements Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Bump requirements in plugins Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Handle nested Annotated Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Leave TODO re: strucutured dataset type castability Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Remove mention to flyteidl@union_type in doc-requirements Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Linting Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Use tempfile.mkdtemp to create a temporary directory for local data persistence. Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Revert "Use tempfile.mkdtemp to create a temporary directory for local data persistence." This reverts commit 861b157. Signed-off-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local> * Force temporary file to not be deleted in test Signed-off-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local> * Regenerate papermill dev-requirements Signed-off-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local> * Remove duplicate code Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> * Put a lower bound on the pip version installed in CI Signed-off-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Kevin Su <pingsutw@apache.org> Co-authored-by: Emirhan Karagül <emirhan350z@gmail.com> Co-authored-by: Eduardo Apolinario <653394+eapolinario@users.noreply.github.com> Co-authored-by: Eduardo Apolinario <eapolinario@users.noreply.github.com> Co-authored-by: Kevin Su <pingsutw@gmail.com> Co-authored-by: Yee Hing Tong <wild-endeavor@users.noreply.github.com> Co-authored-by: Ketan Umare <16888709+kumare3@users.noreply.github.com> Co-authored-by: Lisa <30621230+aeioulisa@users.noreply.github.com> Co-authored-by: Niels Bantilan <niels.bantilan@gmail.com> Co-authored-by: Stef Nelson-Lindall <bethebunny@gmail.com> Co-authored-by: Snyk bot <github+bot@snyk.io> Co-authored-by: Dan Rammer <hamersaw@protonmail.com> Co-authored-by: bstadlbauer <11799671+bstadlbauer@users.noreply.github.com> Co-authored-by: Bernhard Stadlbauer <bstadlbauer@gmx.net> Co-authored-by: Bernhard Stadlbauer <bstadlbauer@blackshark.ai> Co-authored-by: Samhita Alla <aallasamhita@gmail.com> Co-authored-by: Yuvraj <code@evalsocket.dev> Co-authored-by: Zach Palchick <palchicz@users.noreply.github.com> Co-authored-by: Snyk bot <snyk-bot@snyk.io> Co-authored-by: Kenny Workman <31255434+kennyworkman@users.noreply.github.com> Co-authored-by: ggydush-fn <69013027+ggydush-fn@users.noreply.github.com> Co-authored-by: SmritiSatyanV <94349093+SmritiSatyanV@users.noreply.github.com> Co-authored-by: Matthew Griffin <1matthewgriffin@gmail.com> Co-authored-by: eduardo apolinario <eduardo@eduardos-MacBook-Pro.local> Signed-off-by: Mike Zhong <mzhong@embarkvet.com>

kennyworkman requested review from eapolinario, kumare3 and wild-endeavor as code owners December 1, 2021 16:57

kennyworkman force-pushed the kenny/type-annotation branch from 33621a6 to e9191a8 Compare December 1, 2021 16:58

eapolinario reviewed Dec 1, 2021

View reviewed changes

flytekit/core/type_engine.py Outdated Show resolved Hide resolved

flytekit/core/type_engine.py Outdated Show resolved Hide resolved

flytekit/core/type_engine.py Outdated Show resolved Hide resolved

kennyworkman mentioned this pull request Dec 2, 2021

Kenny/type metadata flyteorg/flytepropeller#370

Merged

8 tasks

kennyworkman force-pushed the kenny/type-annotation branch from 235f01b to 4ba1f99 Compare December 2, 2021 06:18

wild-endeavor reviewed Dec 3, 2021

View reviewed changes

kennyworkman force-pushed the kenny/type-annotation branch 5 times, most recently from 9f316f9 to fe378f5 Compare December 9, 2021 19:39

eapolinario requested changes Dec 10, 2021

View reviewed changes

eapolinario reviewed Dec 13, 2021

View reviewed changes

requirements.txt Outdated Show resolved Hide resolved

flytekit/core/type_engine.py Outdated Show resolved Hide resolved

flytekit/models/types.py Outdated Show resolved Hide resolved

flytekit/models/types.py Show resolved Hide resolved

flytekit/models/types.py Outdated Show resolved Hide resolved

eapolinario reviewed Dec 14, 2021

View reviewed changes

flytekit/core/annotation.py Show resolved Hide resolved

kennyworkman force-pushed the kenny/type-annotation branch from 2809118 to 97f1219 Compare December 30, 2021 19:00

kennyworkman force-pushed the kenny/type-annotation branch 4 times, most recently from f9525ac to 25c5928 Compare February 8, 2022 06:03

kennyworkman added 4 commits February 7, 2022 22:04

feat: support for annotated simple + list

71619aa

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

feat: addition of annotation att to

ac27cc0

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

feat: core obj

1417294

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

feat: proto model

e089b17

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

kennyworkman added 4 commits February 7, 2022 22:04

fix: typing_ext

1a5840f

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

fix: plugin tmp requirements

c0c3073

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

fix: bump requirements

a6a6cb4

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

fix: doc requirements

9df04ea

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

kennyworkman force-pushed the kenny/type-annotation branch 2 times, most recently from e32d0e9 to b0a65f4 Compare February 8, 2022 07:08

kennyworkman added 8 commits February 7, 2022 23:09

fix: whitespace

0ed30b8

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

fix: bump flytekit

5fc72d4

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

fix: numpy version

9f62e9c

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

fix: lint

4dbbac9

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

fix: pandas version

75c91d9

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

fix: bump requirements

183f110

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

fix: test import

d3f4830

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

fix: flake8 lint

854b96f

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

kennyworkman force-pushed the kenny/type-annotation branch from b0a65f4 to 854b96f Compare February 8, 2022 07:09

kennyworkman added 5 commits February 7, 2022 23:21

fix: merge

5173271

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

fix: merge

2ec2d94

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

fix: requirements

956f8e6

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

fix: requirements

19d5642

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

fix: lint

755f31f

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

kennyworkman force-pushed the kenny/type-annotation branch from cdfa0d3 to 755f31f Compare February 8, 2022 07:41

kennyworkman added 3 commits February 8, 2022 09:34

Merge remote-tracking branch 'real/master' into kenny/type-annotation

0c232e8

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

fix: papermill req

4d64acd

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

fix: req

88bc4d0

Signed-off-by: Kenny Workman <kennyworkman@sbcglobal.net>

eapolinario approved these changes Feb 8, 2022

View reviewed changes

eapolinario merged commit d249cb2 into flyteorg:master Feb 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TypeAnnotation #759

TypeAnnotation #759

kennyworkman commented Dec 1, 2021

welcome bot commented Dec 1, 2021

eapolinario left a comment

kennyworkman commented Dec 2, 2021

eapolinario commented Dec 3, 2021

wild-endeavor left a comment

eapolinario left a comment

eapolinario left a comment

eapolinario commented Dec 14, 2021

codecov bot commented Dec 14, 2021 •

edited

eapolinario left a comment

welcome bot commented Feb 8, 2022

TypeAnnotation #759

TypeAnnotation #759

Conversation

kennyworkman commented Dec 1, 2021

TL;DR

Type

Are all requirements met?

Complete description

Tracking Issue

Follow-up issue

welcome bot commented Dec 1, 2021

eapolinario left a comment

Choose a reason for hiding this comment

kennyworkman commented Dec 2, 2021

eapolinario commented Dec 3, 2021

wild-endeavor left a comment

Choose a reason for hiding this comment

eapolinario left a comment

Choose a reason for hiding this comment

eapolinario left a comment

Choose a reason for hiding this comment

eapolinario commented Dec 14, 2021

codecov bot commented Dec 14, 2021 • edited

Codecov Report

eapolinario left a comment

Choose a reason for hiding this comment

welcome bot commented Feb 8, 2022

codecov bot commented Dec 14, 2021 •

edited