Dispatch Metaflow flows to Argo Workflows #992
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
corleyma
reviewed
Mar 23, 2022
oavdeev
reviewed
Mar 23, 2022
Open
corleyma
previously approved these changes
Apr 4, 2022
oavdeev
reviewed
Apr 20, 2022
This was referenced Apr 22, 2022
npow
added a commit
to npow/metaflow
that referenced
this pull request
Jan 12, 2023
* Don't explicitly break py2 support (#962)
* Don't explicitly break py2 support
* Typo
* Typo
* Pass the paths used by the external interpreter to the one launching the Env Escape server (#960)
We were currently not reflecting possible values passed using PYTHONPATH or
programmatically added to the interpreter launching the environment escape server.
* Bump follow-redirects in /metaflow/plugins/cards/ui (#966)
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.7 to 1.14.9.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.14.7...v1.14.9)
---
updated-dependencies:
- dependency-name: follow-redirects
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Adding export of graph to JSON for cli (#955)
* Fix "Too many symbolic links" error when using Conda + Batch on MacOS (#972)
* Fix "Too many symbolic links" error when using Conda + Batch on MacOS
* Ran black
* emit app tag for AWS Batch jobs (#970)
* Bump to 2.5.3 (#974)
* Extension packaging improvements (#959)
* Fixed dual distribution import in some cases; fixed get_plugin_cli extension
If sys.path contains the same directory multiple times, metadata.distributions()
will list a package multiple times which breaks the import mechanism we have. This
is addressed by skipping duplicate packages.
get_plugin_cli for extensions wasn't evaluated late enough therefore
not allowing additional parameters to be added.
* Fix issue with metaflow_extensions and Conda environment
In a Conda environment, it is not possible to re-resolve all the
metaflow_extension packages since they appear as a single directory.
In that case, we resolve on the INFO file to give us the proper
information.
Also fixed the way metaflow_extensions are added to the Conda
environment to avoid leaking more information than needed.
* Validate configuration files for extensions and de-duplicate for internal ns packages
* Update packaging of metaflow_extensions packages
Packaging is now handled directly by extension_support.py and
each distribution/package can define its own suffixes to include.
* Rework the import mechanism for modules contained in metaflow_extensions
This change does away with the deprecated load_module and also improves
handling of loading children modules and _orig modules
* Address comments
* Squash merge of origin/master
* Fix issue when packages were left out
* Merge get_pinned_conda_libs
* Fix issue with __init__.py in non-distribution packages
* Moving card import changes on top of Romain's branch (#973)
* Moving card import changes on top of Romain's branch
- tiny bug fix to extensions.
- Added card related refactor to support mfextinit and regular package.
- removing card packaging related code in decorator + everywhere else.
* Tiny bug fix on top of romain's new changes.
* removing unneccessary logic
* Added a bit more inline documentation and addressed documentation comments
* Properly handle parsing of package requirements
* Fix Pylint errors in some case of metaflow_extensions module aliasing
* Properly handle case of files at the root of PYTHONPATH package
* Tests for Extensions (#978)
* Added tests for extensions. Checking if they work.
* dummy commit to see if things work.
* fix
* debug
* bug fix.
* possible fix.
* tweeking context
* Bug fix to tests.
* dummy commit
* bug fix
* Added extension test to core tests.
- remove seperate gh action
* removing files.
* fix
* added extension test to py3 context
- remove redundant complexity.
Co-authored-by: Valay Dave <valaygaurang@gmail.com>
* pass DEFAULT_AWS_CLIENT_PROVIDER to remote tasks (#982)
* Fixing some hacky plumbing in card test suite (#967)
* Fixing some hacky plumbing for card tests.
- Added `--file` arg to card list cli
- Using files to get the information written by cli
- changes to test to use files instead of stdout
- Piping stderrors to stderrr to capture card not found errors.
* Removing `\n` from all card tests assertions
- `\n` was there because earlier we read stdout.
- Now since we read files, it will not be needed.
* Simplify mflog (#979)
* Fix extension root determination in some cases + card tweaks (#989)
* Use importlib_metadata 2.8.3 for Python 3.4, 3.5 but 4.8.3 for Python 3.6+ (#988)
* Use importlib_metadata 2.8.3 for Python 3.4 and 3.5 but 4.8.3 for Python 3.6+
This is required because in importlib_metadata 3.4.0, a field called `_normalized_name`
was introduced and is relied on by later versions of importlib_metadata. When
importlib_metadata looks for distributions, it queries all registered/installed
importlib_metadata and if an older version (like 2.8.3) returns something that
does not have this field, it causes a crash.
* Add __init__.py files to make the vendored package non namespace
* Allow configuring the root directory where artifacts go when pulling from S3. (#991)
Co-authored-by: kgullikson <kgullikson@sparkcognition.com>
* Add an option to pass a `role` argument to S3() to provide a role (#987)
* Add an option to pass a `role` argument to S3() to provide a role for the S3 client
* Move to partial to be able to pickle ops
* Address comments
* bump version to 2.5.4 (#993)
* Bug fix for datetime index in default card (#981)
* Bug fix for datetime_indexes.
* ran black
* Bump minimist from 1.2.5 to 1.2.6 in /metaflow/plugins/cards/ui (#995)
Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6.
- [Release notes](https://github.com/substack/minimist/releases)
- [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6)
---
updated-dependencies:
- dependency-name: minimist
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Fix black - set upper bound for click version (#1006)
* Add tags to `current` singleton (#1019)
* first attempt
* fix _set_env param typo
* add assertions for tags to current singleton test
* Dispatch Metaflow flows to Argo Workflows (#992)
* Dispatch Metaflow flows to Argo Workflows
* Remove spurious files
* Make black happy
* Make black happy again
* Remove spurious TODOs
* Handle non-numeric task ids
* Fix typos
* don't actively automount service account tokens
* make black happy
* Update @kubernetes
* remove spurious commits
* fix issue with hyphens
* raise memory for tests
* change limits to requests
* pin back test resources
* update refresh timeout
* stylistic nits
* Add lint check
* make black happy
* support gpus
* drop extra bracket
* fix
* Skip sleep if not retryting S3 operations (#1001)
* Don't sleep if not retrying S3 operations
* Remove unused env variable
* VSCode is too smart
* Bump minor version for release (#1022)
* Alternate way of adding metadata to cloned tasks (#1003)
* Alternate way of adding metadata to cloned tasks
This adds it on the client side
* Clean up metadata passing.
Also optimize use within the client
* Typo
* Fix black test
* Keep black happy
* Added tests
* Ignore namespace when getting origin task
* Bump cross-fetch from 3.1.4 to 3.1.5 in /metaflow/plugins/cards/ui (#1027)
Bumps [cross-fetch](https://github.com/lquixada/cross-fetch) from 3.1.4 to 3.1.5.
- [Release notes](https://github.com/lquixada/cross-fetch/releases)
- [Commits](https://github.com/lquixada/cross-fetch/compare/v3.1.4...v3.1.5)
---
updated-dependencies:
- dependency-name: cross-fetch
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fixed bug from #1023 (#1025)
* current.pathspec to return None when used outside Flow (#1033)
* current.pathspec to return None when used outside Flow
* formatted with black
* Fixing bug in the `card list` command (#1044)
- echo_always defaulted to stderr :(
* Enable MinIO/EMC-ECS as blob stores for Metaflow (#1045)
* Enable EMCECS
* Enable MinIO & EMC-ECS blob stores for Metaflow
* fix InvalidRange issue
* [tag-mutation-project] Stash system info in task metadata (#1039)
* stash system info as metadata (in addition to system_tags today)
* improve
* avoid parsing on ":" for user id
* remove date - it's not well defined and created_at is available
* python_version is in metadata now, and can vary between orig and resume runs
* address CR comments
* add the comments
* Address issues with S3 get and ranges (#1034)
* Address issues with S3 get and ranges
This addresses two issues:
- get and get_many would not properly populate the range_info in S3Object
- get_many on different ranges of the same file would return incorrect results
* Add one more test (merge with Netflix internal tests)
* Put Run() test behind a check
* Address comments
* Fix _new_task calling bug in LocalMetadataProvider (#1046)
* Release 2.6.1 (#1047)
* Support default secrets (#1048)
* Run id file should be written prior to execution when resuming (#1051)
* Run id file should be written prior to execution when resuming
* separate run id test so we can exclude from batch
* release 2.6.2 (#1052)
* Fix instance metadata calls for IMDSV2 (#1053)
* bump version to 2.6.3 (#1054)
* Small reorder of imports to make things more consistent (#1055)
* [tag-mutation-project] Local metadata provider - object read paths to return ancestral Run tags (#1043)
* wip
* Fix nits
* typo fix. (#1068)
* This is the Jackie's tag changes with an additional test and a fix. (#1063)
* rebase on master
* add replace_many as candidate subcommand
* updates from some CR suggestions
* propagate opt rename
* more changes follow CR
* improve consistency guarantees of local metadata tag mutations (retries if a race was lost)
* write comment
* use replace-many logic to back "tag replace"
* finalize client API for tag operations
* fix retry logic
* Add task.tags == run.tags check in test
* [tag-mutation-project] Add tag mutation support to ServiceMetadataProvider
* fix version
* Enable TagMutationTest in batch and k8s
* Bogus commit
* minor UX fixes on CLI
* Fully support CliCheck for TagMutationTest
* add tag and tag set validation to local metadata code path
Validate tags on run / resume paths too
* flat nor flatten on CLI
* Logs really do come via stdout - related to CliCheck refactor
* argo and step functions - validate tags as soon as possible (before other user messaging)
* no capture_output pre 3.7
* Add another test for tag mutation
* Properly pass down the metadata provider
There were two cases when the metadata provider was not properly
passed down to the client:
- when calling the client within a flow
- when invoking the `tag` command
This fixes both issues and also allows the user to introspect
the metadata provider and datastore being used for the flow
* Do not make public metadata and datastore in current object
* Typo
Co-authored-by: Jackie Tung <jackie@outerbounds.co>
Co-authored-by: jackie-ob <104396218+jackie-ob@users.noreply.github.com>
* Add two options to resume: the ability to specify a run-id and the ab… (#1059)
* Add two options to resume: the ability to specify a run-id and the ability to only clone tasks
The scenario for this use is mainly for external schedulers that are
capable of "resuming" failed runs by only re-executing the ones that failed
or didn't execute the first time. We need a way for Metaflow to "clone" the
tasks that are not re-executed.
* Small formatting fix
* Made clone-only resume possibly re-entrant
* Addressed comments
* Clarify the reentrant behavior of resume
* Improve sidecar message handling (#1057)
* Improve sidecar message handling
Several changes:
- allow side-passing of a static "context" to the sidecar process to keep
messages as small as possible
- clean up the interfaces and make it clearer what is what (ie: there is
an emitter inside the process and then a sidecar that runs outside)
- better error checking
- better handling of shutdown (sidecars will now properly get a shutdown
message giving them the opportunity to clean up -- this may add a bit
of latency at the end of the program execution.
Also improved message when plugins don't load in case of an issue with
Metaflow extensions
* Add the possibility to pass additional context to sidecars
This allows more information to be provided after the sidecar is
created but still maintains small message sizes
* Better handling of initial context message for sidecars
In the previous implementation, a failure to send the initial context
message could crash the flow. This change properly handles that message
like any other message (so has no impact on the flow itself). It also
implements better retry policies to send the context. Finally, this change
also improves error handling: if a message fails to send, in the previous
implementation, it was likely that the next successful one would be
an invalid json.
* Improve handling of corner conditions on message send
* Restart sidecar faster when sidecar process dies
* Properly send context after sidecar restart
* Remove the specific context from monitors/loggers
Subclasses can now handle contexts directly in the subclass.
Refactoring also cleaned up what a sidecar and a sidecar worker
do by refactoring some of the code out.
* Typo: fix names
* Minor typos for sidecar changes
* Clean up sidecars
* More cleanups for sidecars
* Addressed final comments for sidecars
* Small typo in NullEventLogger.send (#1071)
Pushing directly per our conversation with Savin
* Bump version to 2.7.0
* Change behavior of env escape when module is present (#1072)
When a module configured in the env escape was present in the local
environment, we would raise an error. We now print a message and use
the module present in the environment (not the environment escape).
This allows the env escape modules to provide a "fallback" in case the
module is not present but to still use the local module if present
* Bump version to 2.7.1 (#1073)
* Fix an issue with the environment escape server directory (#1074)
The environment escape server would launch in the cwd. This could
cause issues if, for example, a `metaflow` directory existed in that
cwd (because it would then take precedence over the actual metaflow.
To remediate this situation, the server is launched in the configuration
directory (where we know what there is).
Also took care of a quick annoyance of having to create an empty overrides.py
file for everything. This is now no longer required
* Support session_vars and client_params in S3() (#1069)
* Support session_vars and client_params in S3()
This allows the user to pass down session variables and client parameters
to set when getting a S3 client.
* Addressed comments about default configuration
* Support M1 Macs (#1077)
* changes to @kubernetes for sandbox (#1079)
* Changed error to warning in AWS retry (#1081)
Co-authored-by: Preetam Joshi <preetamj@netflix.com>
* bump to 2.7.2 (#1083)
* Fixed s3util test (#1084)
* Fixing s3util test caset
* added comment
Co-authored-by: Preetam Joshi <preetamj@netflix.com>
* Bump svelte from 3.46.1 to 3.49.0 in /metaflow/plugins/cards/ui (#1086)
Bumps [svelte](https://github.com/sveltejs/svelte) from 3.46.1 to 3.49.0.
- [Release notes](https://github.com/sveltejs/svelte/releases)
- [Changelog](https://github.com/sveltejs/svelte/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sveltejs/svelte/compare/v3.46.1...v3.49.0)
---
updated-dependencies:
- dependency-name: svelte
dependency-type: direct:development
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Metadata version check flag (#1088)
* Bump terser from 5.10.0 to 5.14.2 in /metaflow/plugins/cards/ui (#1090)
Bumps [terser](https://github.com/terser/terser) from 5.10.0 to 5.14.2.
- [Release notes](https://github.com/terser/terser/releases)
- [Changelog](https://github.com/terser/terser/blob/master/CHANGELOG.md)
- [Commits](https://github.com/terser/terser/commits)
---
updated-dependencies:
- dependency-name: terser
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix r CI deps (#1092)
* fix r CI deps
* fix fractional resoure handling for batch (#1089)
* bump version (#1093)
* Fix docstrings for the API reference (no functional changes!) (#1076)
Make the docstring format in public APIs compliant with the new API reference framework
* Move a sys.path modification in s3op to __main__ (#1095)
In its current location, this could modify the sys.path of the current
running metaflow which could have nefarious consequences with the escape
hatch which uses sys.path to determine the outside environment's python path.
The following scenario would cause issues:
- metaflow is installed in the usual path on the system
- a conda environment was manually bootstrapped from a directory A.
+ at this point, sys.path starts with `A` and then contains the other
system includes
+ s3op.py is imported at some point by the Conda installer when it calls `get_many`
+ this modifies sys.path to insert, at the beginning, the parent of Metaflow; so
in this case, sys.path looks something like
['/apps/python3/lib/python3.7/site-packages', 'A', '/apps/python3/lib/python3.7/site-packages'...]
+ when the escape hatch trampolines are created, this sys.path is used to determine what the
sys.path for the outside interpreter is.
+ in A, we create:
* INFO
* metaflow
* metaflow_extensions
which properly describe the installation of metaflow
- when the escape hatch client runs, it runs in the conda environment and uses metaflow
created in A.
- when the client wants to start the server, this is where we run into issues because, at this
point, the server will use the PYTHONPATH which starts with '/apps/python3/lib/python3.7/site-packages'
in which it will find metaflow. It will therefore use that metaflow (which is the same
as the one linked in A) to start the server. This runs into issues though because A
is also in PYTHONPATH and so the extension support loader will also try to load `A/metaflow_extensions`.
This will cause issues if multiple extensions are installed there (it will complain about duplicate
configurations for example. The `INFO` file typically used to solve this problem is not read as it was
not present for the TL metaflow.
This patch simply moves the modification of sys.path to where it is actually needed and avoids polluting
sys.path when the module is simply included (and not called as a script).
* Airflow Support (#1094)
* Airflow on Kubernetes minus Foreachs.
- Support for all metaflow construct without foreach and sensors
Squashed commit of the following:
commit ef8b1e3768695bc4d3375a947ab1da9c6520bcf1
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jul 29 01:06:26 2022 +0000
Removed sernsors and banned foreach's
commit 8d517c4fecc6568777ad03eca81aaacfa3e91156
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jul 29 00:59:01 2022 +0000
commiting k8s related file from master.
commit a7e1ecdbf7b8b8d1cc21321cc8e196053f8305e4
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jul 29 00:54:45 2022 +0000
Uncommented code for foreach support with k8s
KubernetesPodOperator version 4.2.0 renamed `resources` to
`container_resources`
- Check : (https://github.com/apache/airflow/pull/24673) /
- (https://github.com/apache/airflow/commit/45f4290712f5f779e57034f81dbaab5d77d5de85)
This was done because `KubernetesPodOperator` didn't play nice with dynamic task mapping and they had to deprecate the `resources` argument. Hence the below codepath checks for the version of `KubernetesPodOperator`
and then sets the argument. If the version < 4.2.0 then we set the argument as `resources`.
If it is > 4.2.0 then we set the argument as `container_resources`
The `resources` argument of KuberentesPodOperator is going to be deprecated soon in the future.
So we will only use it for `KuberentesPodOperator` version < 4.2.0
The `resources` argument will also not work for foreach's.
commit 2719f5d792ada91e3ae0af6f1a9a0c7d90f74660
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:31:58 2022 +0000
nit fixes :
- fixing comments.
- refactor some variable/function names.
commit 2079293fbba0d3d862476a7d67b36af8a3389342
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:14:53 2022 +0000
change `token` to `production_token`
commit 14aad5ff717418e4183a88fa84b2f5e5bb13927a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:11:56 2022 +0000
Refactored import Airflow Sensors.
commit b1472d5f7a629024ca45e8b83700400d02a4d455
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:08:41 2022 +0000
new comment on `startup_timeout_seconds` env var.
commit 6d81b758e8f06911258d26f790029f557488a0d7
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:06:09 2022 +0000
Removing traces of `@airflow_schedule_interval`
commit 0673db7475b22f3ce17c2680fc0a7c4271b5c946
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Jul 14 12:43:08 2022 -0700
Foreach polish (valayDave/metaflow#62)
* Removing unused imports
* Added validation logic for airflow version numbers with foreaches
* Removed `airflow_schedule_interval` decorator.
* Added production/deployment token related changes
- Uses s3 as a backend to store the production token
- Token used for avoiding nameclashes
- token stored via `FlowDatastore`
* Graph type validation for airflow foreachs
- Airflow foreachs only support single node fanout.
- validation invalidates graphs with nested foreachs
* Added configuration about startup_timeout.
* Added final todo on `resources` argument of k8sOp
- added a commented code block
- it needs to be uncommented when airflow releasese the patch for the op
- Code seems feature complete keeping aside airflow patch
commit 4b2dd1211fe2daeb76e29e4084f21e96b10cdae9
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Jul 7 19:33:07 2022 +0000
Removed retries from user-defaults.
commit 0e87a97fea15ba3aaa6d4228b141bd796b767c43
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Jul 6 16:29:33 2022 +0000
updated pod startup time
commit fce2bd263f368dbb78a34ac71f64e13c89277222
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Jun 29 18:44:11 2022 +0000
Adding default 1 retry for any airflow worker.
commit 5ef6bbcde51b1f4923a192291ed0e07d07ec7321
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 27 01:22:42 2022 +0000
Airflow Foreach Integration
- Simple one node foreach-join support as gaurenteed by airflow
- Fixed env variable setting issue
- introduced MetaflowKuberentesOperator
- Created a new operator to allow smootness in plumbing xcom values
- Some todos
commit d319fa915c558d82f1d127736ce34d3ae0da521d
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 24 21:12:09 2022 +0000
simplifying run-id macro.
commit 0ffc813b1c4e6ba0103be51520f42d191371741a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 24 11:51:42 2022 -0700
Refactored parameter macro settings. (valayDave/metaflow#60)
commit a3a495077f34183d706c0edbe56d6213766bf5f6
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 24 02:05:57 2022 +0000
added comment on need for `start_date`
commit a3147bee08a260aa78ab2fb14c6232bfab2c2dec
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Jun 21 06:03:56 2022 +0000
Refactored an `id_creator` method.
commit 04d7f207ef2dae0ce2da2ec37163ac871f4517bc
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Jun 21 05:52:05 2022 +0000
refactor :
-`RUN_ID_LEN` to `RUN_HASH_ID_LEN`
- `TASK_ID_LEN` to `TASK_ID_HASH_LEN`
commit cde4605cd57ad9214f5a6afd7f58fe4c377e09e2
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Jun 21 05:48:55 2022 +0000
refactored an error string
commit 11458188b6c59d044fca0dd2d1f5024ec84f6488
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 20 22:42:36 2022 -0700
addressing savins comments. (#59)
- Added many adhoc changes based for some comments.
- Integrated secrets and `KUBERNETES_SECRETS`
- cleaned up parameter setting
- cleaned up setting of scheduling interval
- renamed `AIRFLOW_TASK_ID_TEMPLATE_VALUE` to `AIRFLOW_TASK_ID`
- renamed `AirflowSensorDecorator.compile` to `AirflowSensorDecorator.validate`
- Checking if dagfile and flow file are same.
- fixing variable names.
- checking out `kubernetes_decorator.py` from master (6441ed5)
- bug fixing secret setting in airflow.
- simplified parameter type parsing logic
- refactoring airflow argument parsing code.
commit 83b20a7c6a13b3aedb7e603e139f07f0ef2fb646
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 13 14:02:57 2022 -0700
Addressing Final comments. (#57)
- Added dag-run timeout.
- airflow related scheduling checks in decorator.
- Auto naming sensors if no name is provided
- Annotations to k8s operators
- fix: argument serialization for `DAG` arguments (method names refactored like `to_dict` became `serialize`)
- annotation bug fix
- setting`workflow-timeout` for only scheduled dags
commit 4931f9c84e6a1d20fc3ecb41cf138b72e5dee629
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:50:49 2022 +0000
k8s bug fix
commit 200ae8ed4a00028f094281f73a939e7a4dcdf83a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:39:50 2022 +0000
removed un-used function
commit 70e285e9a7cfbec71fc293508a62c96f33562a01
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:38:37 2022 +0000
Removed unused `sanitize_label` function
commit 84fc622d8b11e718a849b2e2d91ceb3ea69917e6
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:37:34 2022 +0000
GPU support added + container naming same as argo
commit c92280d8796ec12b4ff17fa2ff3c736c7244f39c
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:25:17 2022 +0000
Refactored sensors to different files + bug fix
- bug caused due `util.compress_list`.
- The function doesn't play nice with strings with variety of characters.
- Ensured that exceptions are handled appropriately.
- Made new file for each sensor under `airflow.sensors` module.
commit b72a1dcf0dbbbcb814581d92738fd27ec31ef673
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Jun 4 01:41:49 2022 +0000
ran black.
commit 558c82f65b383ed0d61ded6bc80326471e284550
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 18:32:48 2022 -0700
Moving information from airflow_utils to compiler (#56)
- commenting todos to organize unfinished changes.
- some environment variables set via`V1EnvVar`
- `client.V1ObjectFieldSelector` mapped env vars were not working in json form
- Moving k8s operator import into its own function.
- env vars moved.
commit 9bb5f638792a671164ec95891e97f599e9a3385f
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 18:06:03 2022 +0000
added mising Run-id prefixes to variables.
- merged `hash` and `dash_connect` filters.
commit 37b5e6a9d8ca93cc91244c8d77c7d4f61280ba59
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 18:00:22 2022 +0000
nit fix : variable name change.
commit 660756f952ebd92ba1e26d7f908b81036c31ff10
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 17:58:34 2022 +0000
nit fixes to dag.py's templating variables.
commit 1202f5bc92f76df52b5957f11c8574cadfa62196
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 17:56:53 2022 +0000
Fixed defaults passing
- Addressed comments for airflow.py
commit b9387dd428c1a37f9a3bfe2c72cab475da708c02
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 17:52:24 2022 +0000
Following Changes:
- Refactors setting scheduling interval
- refactor dag file creating function
- refactored is_active to is_paused_upon_creation
- removed catchup
commit 054e3f389febc6c447494a1dedb01228f5f5650f
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 17:33:25 2022 +0000
Multiple Changes based on comments:
1. refactored `create_k8s_args` into _to_job
2. Addressed comments for snake casing
3. refactored `attrs` for simplicity.
4. refactored `metaflow_parameters` to `parameters`.
5. Refactored setting of `input_paths`
commit d481b2fca7914b6b657a69af407cfe1a894a46dc
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 16:42:24 2022 +0000
Removed Sensor metadata extraction.
commit d8e6ec044ef8c285d7fbe1b83c10c07d51c063e3
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 16:30:34 2022 +0000
porting savin's comments
- next changes : addressing comments.
commit 3f2353a647e53bc240e28792769c42a71ea8f8c9
Merge: d370ffb c1ff469
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Jul 28 23:52:16 2022 +0000
Merge branch 'master' into airflow
commit d370ffb248411ad4675f9d55de709dbd75d3806e
Merge: a82f144 e4eb751
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Jul 14 19:38:48 2022 +0000
Merge branch 'master' into airflow
commit a82f1447b414171fc5611758cb6c12fc692f55f9
Merge: bdb1f0d 6f097e3
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Jul 13 00:35:49 2022 +0000
Merge branch 'master' into airflow
commit bdb1f0dd248d01318d4a493c75b6f54248c7be64
Merge: 8511215 f9a4968
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Jun 29 18:44:51 2022 +0000
Merge branch 'master' into airflow
commit 85112158cd352cb7de95a2262c011c6f43d98283
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Jun 21 02:53:11 2022 +0000
Bug fix from master merge.
commit 90c06f12bb14eda51c6a641766c5f67d6763abaa
Merge: 0fb73af 6441ed5
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 20 21:20:20 2022 +0000
Merge branch 'master' into airflow
commit 0fb73af8af9fca2875261e3bdd305a0daab1b229
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Jun 4 00:53:10 2022 +0000
squashing bugs after changes from master.
commit 09c6ba779f6b1b6ef1d7ed5b1bb2be70ec76575d
Merge: 7bdf662 ffff49b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Jun 4 00:20:38 2022 +0000
Merge branch 'master' into af-mmr
commit 7bdf662e14966b929b8369c65d5bd3bbe5741937
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon May 16 17:42:38 2022 -0700
Airflow sensor api (#3)
* Fixed run-id setting
- Change gaurentees that multiple dags triggered at same moment have unique run-id
* added allow multiple in `Decorator` class
* Airflow sensor integration.
>> support added for :
- ExternalTaskSensor
- S3KeySensor
- SqlSensor
>> sensors allow multiple decorators
>> sensors accept those arguments which are supported by airflow
* Added `@airflow_schedule_interval` decorator
* Fixing bug run-id related in env variable setting.
commit 2604a29452e794354cf4c612f48bae7cf45856ee
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 21 18:26:59 2022 +0000
Addressed comments.
commit 584e88b679fed7d6eec8ce564bf3707359170568
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:33:55 2022 +0000
fixed printing bug
commit 169ac1535e5567149d94749ddaf70264e882d62c
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:30:59 2022 +0000
Option help bug fix.
commit 6f8489bcc3bd715b65d8a8554a0f3932dc78c6f5
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:25:54 2022 +0000
variable renamemetaflow_specific_args
commit 0c779abcd1d9574878da6de8183461b53e0da366
Merge: d299b13 5a61508
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:23:10 2022 +0000
Merge branch 'airflow-tests' into airflow
commit 5a61508e61583b567ef8d3fea04e049d74a6d973
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:22:54 2022 +0000
Removing un-used code / resolved-todos.
commit d030830f2543f489a1c4ebd17da1b47942f041d6
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:06:03 2022 +0000
ran black,
commit 2d1fc06e41cbe45ccfd46e03bc87b09c7a78da45
Merge: f2cb319 7921d13
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:04:19 2022 +0000
Merge branch 'master' into airflow-tests
commit d299b13ce38d027ab27ce23c9bbcc0f43b222cfa
Merge: f2cb319 7921d13
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:02:37 2022 +0000
Merge branch 'master' into airflow
commit f2cb3197725f11520da0d49cbeef8de215c243eb
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 02:54:03 2022 +0000
reverting change.
commit 05b9db9cf0fe8b40873b2b74e203b4fc82e7fea4
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 02:47:41 2022 +0000
3 changes:
- Removing s3 dep
- remove uesless import
- added `deployed_on` in dag file template
commit c6afba95f5ec05acf7f33fd3228cffd784556e3b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Apr 15 22:50:52 2022 +0000
Fixed passing secrets with kubernetes.
commit c3ce7e9faa5f7a23d309e2f66f778dbca85df22a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Apr 15 22:04:22 2022 +0000
Refactored code .
- removed compute/k8s.py
- Moved k8s code to airflow_compiler.py
- ran isort to airflow_compiler.py
commit d1c343dbbffbddbebd2aeda26d6846e595144e0b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Apr 15 18:02:25 2022 +0000
Added validations about:
- un-supported decorators
- foreach
Changed where validations are done to not save the package.
commit 7b19f8e66e278c75d836daf6a1c7ed2c607417ce
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Apr 15 03:34:26 2022 +0000
Fixing mf log related bug
- No double logging on metaflow.
commit 4d1f6bf9bb32868c949d8c103c8fe44ea41b3f13
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 03:10:51 2022 +0000
Removed usless code WRT project decorator.
commit 5ad9a3949e351b0ac13f11df13446953932e8ffc
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 03:03:19 2022 +0000
Remove readme.
commit 60cb6a79404efe2bcf9bf9a118a68f0b98c7d771
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 03:02:38 2022 +0000
Made file path required arguement.
commit 9f0dc1b2e01ee04b05620630f3a0ec04fe873a31
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 03:01:07 2022 +0000
changed `--is-active`->`--is-paused-upon-creation`
- dags are active by default.
commit 5b98f937a62ee74de8aed8b0efde5045a28f068b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 02:55:46 2022 +0000
shortened length of run-id and task-id hashes.
commit e53426eaa4b156e8bd70ae7510c2e7c66745d101
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 02:41:32 2022 +0000
Removing un-used args.
commit 72cbbfc7424f9be415c22d9144b16a0953f15295
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 02:39:59 2022 +0000
Moved exceptions to airflow compiler
commit b2970ddaa86c393c8abb7f203f6507c386ecbe00
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 02:33:02 2022 +0000
Changes based on PR comments:
- removed airflow xcom push file , moved to decorator code
- removed prefix configuration
- nit fixes.
commit 9e622bac5a75eb9e7a6594d8fa0e47f076634b44
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Apr 11 20:39:00 2022 +0000
Removing un-used code paths + code cleanup
commit 7425f62cff2c9128eea785223ddeb40fa2d8f503
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Apr 11 19:45:04 2022 +0000
Fixing bug fix in schedule.
commit eb775cbadd1d2d2c90f160a95a0f42c8ff0d7f4c
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Apr 10 02:52:59 2022 +0000
Bug fixes WRT Kubernetes secrets + k8s deployments.
- Fixing some error messages.
- Added some comments.
commit 04c92b92c312a4789d3c1e156f61ef57b08dba9f
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Apr 10 01:20:53 2022 +0000
Added secrets support.
commit 4a0a85dff77327640233767e567aee2b379ac13e
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Apr 10 00:11:46 2022 +0000
Bug fix.
commit af91099c0a30c26b58d58696a3ef697ec49a8503
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Apr 10 00:03:34 2022 +0000
bug fix.
commit c17f04a253dfe6118e2779db79da9669aa2fcef2
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Apr 9 23:55:41 2022 +0000
Bug fix in active defaults.
commit 0d372361297857076df6af235d1de7005ac1544a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Apr 9 23:54:02 2022 +0000
@project, @schedule, default active dag support.
- Added a flag to allow setting dag as active on creation
- Airflow compatible schedule interval
- Project name fixes.
commit 5c97b15cb11b5e8279befc5b14c239463750e9b7
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 7 21:15:18 2022 +0000
Max workers and worker pool support.
commit 9c973f2f44c3cb3a98e3e63f6e4dcef898bc8bf2
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 7 19:34:33 2022 +0000
Adding exceptions for missing features.
commit 2a946e2f083a34b4b6ed84c70aebf96b084ee8a2
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Mar 28 19:34:11 2022 +0000
2 changes :
- removed hacky line
- added support to directly throw dags in s3.
commit e0772ec1bad473482c6fd19f8c5e8b9845303c0a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Mar 23 22:38:20 2022 +0000
fixing bugs in service account setting
commit 874b94aeeabc664f12551864eff9d8fdc24dc37b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Mar 20 23:49:15 2022 +0000
Added support for Branching with Airflow
- remove `next` function in `AirflowTask`
- `AirflowTask`s has no knowledge of next tasks.
- removed todos + added some todos
- Graph construction on airflow side using graph_structure datastructure.
- graph_structure comes from`FlowGraph.output_steps()[1]`
commit 8e9f649bd8c51171c38a1e5af70a44a85e7009ca
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Mar 20 02:33:04 2022 +0000
Added hacky line
commit fd5db04cf0a81b14efda5eaf40cd9227e2bac0d3
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Mar 20 02:06:38 2022 +0000
Removed hacky line.
commit 5b23eb7d8446bef71246d853b11edafa93c6ef95
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Mar 20 01:44:57 2022 +0000
Added support for Parameters.
- Supporting int, str, bool, float, JSONType
commit c9378e9b284657357ad2997f2b492bc2f4aaefac
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Mar 20 00:14:10 2022 +0000
Removed todos + added some validation logic.
commit 7250a44e1dea1da3464f6f71d0c5188bd314275a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Mar 19 23:45:15 2022 +0000
Fixing logs related change from master.
commit d125978619ab666dcf96db330acdca40f41b7114
Merge: 8cdac53 7e210a2
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Mar 19 23:42:48 2022 +0000
Merge branch 'master' into aft-mm
commit 8cdac53dd32648455e36955badb8e0ef7b95a2b3
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Mar 19 23:36:47 2022 +0000
making changes sync with master
commit 5a93d9f5198c360b2a84ab13a86496986850953c
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Mar 19 23:29:47 2022 +0000
Fixed bug when using catch + retry
commit 62bc8dff68a6171b3b4222075a8e8ac109f65b4c
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Mar 19 22:58:37 2022 +0000
Changed retry setting.
commit 563a20036a2dfcc48101f680f29d4917d53aa247
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Mar 19 22:42:57 2022 +0000
Fixed setting `task_id` :
- switch task-id from airflow job is to hash to "runid/stepname"
- refactor xcom setting variables
- added comments
commit e2a1e502221dc603385263c82e2c068b9f055188
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Mar 19 17:51:59 2022 +0000
setting retry logic.
commit a697b56052210c8f009b68772c902bbf77713202
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Mar 17 01:02:11 2022 +0000
Nit fix.
commit 68f13beb17c7e73c0dddc142ef2418675a506439
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Mar 16 20:46:19 2022 +0000
Added @schedule support + readme
commit 57bdde54f9ad2c8fe5513dbdb9fd02394664e234
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Mar 15 19:47:06 2022 +0000
Fixed setting run-id / task-id to labels in k8s
- Fixed setting run-id has from cli macro
- added hashing macro to ensure that jinja template set the correct run-id to k8s labels
-
commit 3d6c31917297d0be5f9915b13680fc415ddb4421
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Mar 15 05:39:04 2022 +0000
Got linear workflows working on airflow.
- Still not feature complete as lots of args are still unfilled / lots of unknows
- minor tweek in eks to ensure airflow is k8s compatible.
- passing state around via xcom-push
- HACK : AWS keys are passed in a shady way. : Reverse this soon.
commit db074b8012f76d9d85225a4ceddb2cde8fefa0f4
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Mar 11 12:34:33 2022 -0800
Tweeks
commit a9f0468c4721a2017f1b26eb8edcdd80aaa57203
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Mar 1 17:14:47 2022 -0800
some changes based on savin's comments.
- Added changes to task datastore for different reason : (todo) Decouple these
- Added comments to SFN for reference.
- Airflow DAG is no longer dependent on metaflow
commit f32d089cd3865927bc7510f24ba3418d859410b6
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Feb 23 00:54:17 2022 -0800
First version of dynamic dag compiler.
- Not completely finished code
- Creates generic .py file a JSON that is parsed to create Airflow DAG.
- Currently only boiler plate to make a linear dag but doesn't execute anything.
- Unfinished code.
commit d2def665a86d6a6622d6076882c1c2d54044e773
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Feb 19 14:01:47 2022 -0800
more tweeks.
commit b176311f166788cc3dfc93354a0c5045a4e6a3d4
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Feb 17 09:04:29 2022 -0800
commit 0
- unfinished code.
* Making version compatibility changes.
- Minimum support version to 2.2.0
* Task-id macro related logic refactor
- Done for better version support.
* bug fix: param related task-id setting.
* applied black
* Reverting `decorators.py` to master
* Move sys.path insert earlier in s3op.py (#1098)
* Update setup.py (#1099)
* Fix for env_escape bug when importing local packages (#1100)
* Bump to 2.7.5 (#1102)
* Fix another issue with the escape hatch and paths (#1105)
* Bump to 2.7.6 (#1106)
* add a flag to overwrite config when running metaflow configure sandbox (#1103)
* card dev docs tiny fix. (#1108)
* Fix an issue with get_cards not respecting a Task's ds-root (#1111)
get_cards would not always respect a Task's ds-root leading to cases
where a Task has cards but they cannot be accessed because of an invalid
path.
* Adding support for Azure Blob Storage as a datastore (#1091)
* Azure Storage
* fix BrokenProcessPool import for older pythons
* Fix batch
* add azure configure, cannot validate storage account url after all (catch-22 in configure)
* rename storage account url to AZURE_STORAGE_BLOB_SERVICE_ENDPOINT
* fix indent
* clean up kubernetes config cli, add secrets
* remove signature
* fix fractional resoure handling for batch (#1089)
* bump version (#1093)
* Fix docstrings for the API reference (no functional changes!) (#1076)
Make the docstring format in public APIs compliant with the new API reference framework
* Fix issue with get_pinned_conda_libs and metaflow extensions
* clean up includefile a bit, remove from_env (not used)
* Move a sys.path modification in s3op to __main__ (#1095)
In its current location, this could modify the sys.path of the current
running metaflow which could have nefarious consequences with the escape
hatch which uses sys.path to determine the outside environment's python path.
The following scenario would cause issues:
- metaflow is installed in the usual path on the system
- a conda environment was manually bootstrapped from a directory A.
+ at this point, sys.path starts with `A` and then contains the other
system includes
+ s3op.py is imported at some point by the Conda installer when it calls `get_many`
+ this modifies sys.path to insert, at the beginning, the parent of Metaflow; so
in this case, sys.path looks something like
['/apps/python3/lib/python3.7/site-packages', 'A', '/apps/python3/lib/python3.7/site-packages'...]
+ when the escape hatch trampolines are created, this sys.path is used to determine what the
sys.path for the outside interpreter is.
+ in A, we create:
* INFO
* metaflow
* metaflow_extensions
which properly describe the installation of metaflow
- when the escape hatch client runs, it runs in the conda environment and uses metaflow
created in A.
- when the client wants to start the server, this is where we run into issues because, at this
point, the server will use the PYTHONPATH which starts with '/apps/python3/lib/python3.7/site-packages'
in which it will find metaflow. It will therefore use that metaflow (which is the same
as the one linked in A) to start the server. This runs into issues though because A
is also in PYTHONPATH and so the extension support loader will also try to load `A/metaflow_extensions`.
This will cause issues if multiple extensions are installed there (it will complain about duplicate
configurations for example. The `INFO` file typically used to solve this problem is not read as it was
not present for the TL metaflow.
This patch simply moves the modification of sys.path to where it is actually needed and avoids polluting
sys.path when the module is simply included (and not called as a script).
* Airflow Support (#1094)
* Airflow on Kubernetes minus Foreachs.
- Support for all metaflow construct without foreach and sensors
Squashed commit of the following:
commit ef8b1e3768695bc4d3375a947ab1da9c6520bcf1
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jul 29 01:06:26 2022 +0000
Removed sernsors and banned foreach's
commit 8d517c4fecc6568777ad03eca81aaacfa3e91156
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jul 29 00:59:01 2022 +0000
commiting k8s related file from master.
commit a7e1ecdbf7b8b8d1cc21321cc8e196053f8305e4
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jul 29 00:54:45 2022 +0000
Uncommented code for foreach support with k8s
KubernetesPodOperator version 4.2.0 renamed `resources` to
`container_resources`
- Check : (https://github.com/apache/airflow/pull/24673) /
- (https://github.com/apache/airflow/commit/45f4290712f5f779e57034f81dbaab5d77d5de85)
This was done because `KubernetesPodOperator` didn't play nice with dynamic task mapping and they had to deprecate the `resources` argument. Hence the below codepath checks for the version of `KubernetesPodOperator`
and then sets the argument. If the version < 4.2.0 then we set the argument as `resources`.
If it is > 4.2.0 then we set the argument as `container_resources`
The `resources` argument of KuberentesPodOperator is going to be deprecated soon in the future.
So we will only use it for `KuberentesPodOperator` version < 4.2.0
The `resources` argument will also not work for foreach's.
commit 2719f5d792ada91e3ae0af6f1a9a0c7d90f74660
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:31:58 2022 +0000
nit fixes :
- fixing comments.
- refactor some variable/function names.
commit 2079293fbba0d3d862476a7d67b36af8a3389342
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:14:53 2022 +0000
change `token` to `production_token`
commit 14aad5ff717418e4183a88fa84b2f5e5bb13927a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:11:56 2022 +0000
Refactored import Airflow Sensors.
commit b1472d5f7a629024ca45e8b83700400d02a4d455
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:08:41 2022 +0000
new comment on `startup_timeout_seconds` env var.
commit 6d81b758e8f06911258d26f790029f557488a0d7
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:06:09 2022 +0000
Removing traces of `@airflow_schedule_interval`
commit 0673db7475b22f3ce17c2680fc0a7c4271b5c946
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Jul 14 12:43:08 2022 -0700
Foreach polish (valayDave/metaflow#62)
* Removing unused imports
* Added validation logic for airflow version numbers with foreaches
* Removed `airflow_schedule_interval` decorator.
* Added production/deployment token related changes
- Uses s3 as a backend to store the production token
- Token used for avoiding nameclashes
- token stored via `FlowDatastore`
* Graph type validation for airflow foreachs
- Airflow foreachs only support single node fanout.
- validation invalidates graphs with nested foreachs
* Added configuration about startup_timeout.
* Added final todo on `resources` argument of k8sOp
- added a commented code block
- it needs to be uncommented when airflow releasese the patch for the op
- Code seems feature complete keeping aside airflow patch
commit 4b2dd1211fe2daeb76e29e4084f21e96b10cdae9
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Jul 7 19:33:07 2022 +0000
Removed retries from user-defaults.
commit 0e87a97fea15ba3aaa6d4228b141bd796b767c43
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Jul 6 16:29:33 2022 +0000
updated pod startup time
commit fce2bd263f368dbb78a34ac71f64e13c89277222
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Jun 29 18:44:11 2022 +0000
Adding default 1 retry for any airflow worker.
commit 5ef6bbcde51b1f4923a192291ed0e07d07ec7321
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 27 01:22:42 2022 +0000
Airflow Foreach Integration
- Simple one node foreach-join support as gaurenteed by airflow
- Fixed env variable setting issue
- introduced MetaflowKuberentesOperator
- Created a new operator to allow smootness in plumbing xcom values
- Some todos
commit d319fa915c558d82f1d127736ce34d3ae0da521d
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 24 21:12:09 2022 +0000
simplifying run-id macro.
commit 0ffc813b1c4e6ba0103be51520f42d191371741a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 24 11:51:42 2022 -0700
Refactored parameter macro settings. (valayDave/metaflow#60)
commit a3a495077f34183d706c0edbe56d6213766bf5f6
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 24 02:05:57 2022 +0000
added comment on need for `start_date`
commit a3147bee08a260aa78ab2fb14c6232bfab2c2dec
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Jun 21 06:03:56 2022 +0000
Refactored an `id_creator` method.
commit 04d7f207ef2dae0ce2da2ec37163ac871f4517bc
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Jun 21 05:52:05 2022 +0000
refactor :
-`RUN_ID_LEN` to `RUN_HASH_ID_LEN`
- `TASK_ID_LEN` to `TASK_ID_HASH_LEN`
commit cde4605cd57ad9214f5a6afd7f58fe4c377e09e2
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Jun 21 05:48:55 2022 +0000
refactored an error string
commit 11458188b6c59d044fca0dd2d1f5024ec84f6488
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 20 22:42:36 2022 -0700
addressing savins comments. (#59)
- Added many adhoc changes based for some comments.
- Integrated secrets and `KUBERNETES_SECRETS`
- cleaned up parameter setting
- cleaned up setting of scheduling interval
- renamed `AIRFLOW_TASK_ID_TEMPLATE_VALUE` to `AIRFLOW_TASK_ID`
- renamed `AirflowSensorDecorator.compile` to `AirflowSensorDecorator.validate`
- Checking if dagfile and flow file are same.
- fixing variable names.
- checking out `kubernetes_decorator.py` from master (6441ed5)
- bug fixing secret setting in airflow.
- simplified parameter type parsing logic
- refactoring airflow argument parsing code.
commit 83b20a7c6a13b3aedb7e603e139f07f0ef2fb646
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 13 14:02:57 2022 -0700
Addressing Final comments. (#57)
- Added dag-run timeout.
- airflow related scheduling checks in decorator.
- Auto naming sensors if no name is provided
- Annotations to k8s operators
- fix: argument serialization for `DAG` arguments (method names refactored like `to_dict` became `serialize`)
- annotation bug fix
- setting`workflow-timeout` for only scheduled dags
commit 4931f9c84e6a1d20fc3ecb41cf138b72e5dee629
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:50:49 2022 +0000
k8s bug fix
commit 200ae8ed4a00028f094281f73a939e7a4dcdf83a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:39:50 2022 +0000
removed un-used function
commit 70e285e9a7cfbec71fc293508a62c96f33562a01
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:38:37 2022 +0000
Removed unused `sanitize_label` function
commit 84fc622d8b11e718a849b2e2d91ceb3ea69917e6
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:37:34 2022 +0000
GPU support added + container naming same as argo
commit c92280d8796ec12b4ff17fa2ff3c736c7244f39c
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:25:17 2022 +0000
Refactored sensors to different files + bug fix
- bug caused due `util.compress_list`.
- The function doesn't play nice with strings with variety of characters.
- Ensured that exceptions are handled appropriately.
- Made new file for each sensor under `airflow.sensors` module.
commit b72a1dcf0dbbbcb814581d92738fd27ec31ef673
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Jun 4 01:41:49 2022 +0000
ran black.
commit 558c82f65b383ed0d61ded6bc80326471e284550
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 18:32:48 2022 -0700
Moving information from airflow_utils to compiler (#56)
- commenting todos to organize unfinished changes.
- some environment variables set via`V1EnvVar`
- `client.V1ObjectFieldSelector` mapped env vars were not working in json form
- Moving k8s operator import into its own function.
- env vars moved.
commit 9bb5f638792a671164ec95891e97f599e9a3385f
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 18:06:03 2022 +0000
added mising Run-id prefixes to variables.
- merged `hash` and `dash_connect` filters.
commit 37b5e6a9d8ca93cc91244c8d77c7d4f61280ba59
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 18:00:22 2022 +0000
nit fix : variable name change.
commit 660756f952ebd92ba1e26d7f908b81036c31ff10
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 17:58:34 2022 +0000
nit fixes to dag.py's templating variables.
commit 1202f5bc92f76df52b5957f11c8574cadfa62196
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 17:56:53 2022 +0000
Fixed defaults passing
- Addressed comments for airflow.py
commit b9387dd428c1a37f9a3bfe2c72cab475da708c02
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 17:52:24 2022 +0000
Following Changes:
- Refactors setting scheduling interval
- refactor dag file creating function
- refactored is_active to is_paused_upon_creation
- removed catchup
commit 054e3f389febc6c447494a1dedb01228f5f5650f
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 17:33:25 2022 +0000
Multiple Changes based on comments:
1. refactored `create_k8s_args` into _to_job
2. Addressed comments for snake casing
3. refactored `attrs` for simplicity.
4. refactored `metaflow_parameters` to `parameters`.
5. Refactored setting of `input_paths`
commit d481b2fca7914b6b657a69af407cfe1a894a46dc
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 16:42:24 2022 +0000
Removed Sensor metadata extraction.
commit d8e6ec044ef8c285d7fbe1b83c10c07d51c063e3
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 16:30:34 2022 +0000
porting savin's comments
- next changes : addressing comments.
commit 3f2353a647e53bc240e28792769c42a71ea8f8c9
Merge: d370ffb c1ff469
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Jul 28 23:52:16 2022 +0000
Merge branch 'master' into airflow
commit d370ffb248411ad4675f9d55de709dbd75d3806e
Merge: a82f144 e4eb751
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Jul 14 19:38:48 2022 +0000
Merge branch 'master' into airflow
commit a82f1447b414171fc5611758cb6c12fc692f55f9
Merge: bdb1f0d 6f097e3
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Jul 13 00:35:49 2022 +0000
Merge branch 'master' into airflow
commit bdb1f0dd248d01318d4a493c75b6f54248c7be64
Merge: 8511215 f9a4968
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Jun 29 18:44:51 2022 +0000
Merge branch 'master' into airflow
commit 85112158cd352cb7de95a2262c011c6f43d98283
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Jun 21 02:53:11 2022 +0000
Bug fix from master merge.
commit 90c06f12bb14eda51c6a641766c5f67d6763abaa
Merge: 0fb73af 6441ed5
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 20 21:20:20 2022 +0000
Merge branch 'master' into airflow
commit 0fb73af8af9fca2875261e3bdd305a0daab1b229
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Jun 4 00:53:10 2022 +0000
squashing bugs after changes from master.
commit 09c6ba779f6b1b6ef1d7ed5b1bb2be70ec76575d
Merge: 7bdf662 ffff49b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Jun 4 00:20:38 2022 +0000
Merge branch 'master' into af-mmr
commit 7bdf662e14966b929b8369c65d5bd3bbe5741937
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon May 16 17:42:38 2022 -0700
Airflow sensor api (#3)
* Fixed run-id setting
- Change gaurentees that multiple dags triggered at same moment have unique run-id
* added allow multiple in `Decorator` class
* Airflow sensor integration.
>> support added for :
- ExternalTaskSensor
- S3KeySensor
- SqlSensor
>> sensors allow multiple decorators
>> sensors accept those arguments which are supported by airflow
* Added `@airflow_schedule_interval` decorator
* Fixing bug run-id related in env variable setting.
commit 2604a29452e794354cf4c612f48bae7cf45856ee
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 21 18:26:59 2022 +0000
Addressed comments.
commit 584e88b679fed7d6eec8ce564bf3707359170568
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:33:55 2022 +0000
fixed printing bug
commit 169ac1535e5567149d94749ddaf70264e882d62c
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:30:59 2022 +0000
Option help bug fix.
commit 6f8489bcc3bd715b65d8a8554a0f3932dc78c6f5
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:25:54 2022 +0000
variable renamemetaflow_specific_args
commit 0c779abcd1d9574878da6de8183461b53e0da366
Merge: d299b13 5a61508
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:23:10 2022 +0000
Merge branch 'airflow-tests' into airflow
commit 5a61508e61583b567ef8d3fea04e049d74a6d973
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:22:54 2022 +0000
Removing un-used code / resolved-todos.
commit d030830f2543f489a1c4ebd17da1b47942f041d6
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:06:03 2022 +0000
ran black,
commit 2d1fc06e41cbe45ccfd46e03bc87b09c7a78da45
Merge: f2cb319 7921d13
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:04:19 2022 +0000
Merge branch 'master' into airflow-tests
commit d299b13ce38d027ab27ce23c9bbcc0f43b222cfa
Merge: f2cb319 7921d13
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:02:37 2022 +0000
Merge branch 'master' into airflow
commit f2cb3197725f11520da0d49cbeef8de215c243eb
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 02:54:03 2022 +0000
reverting change.
commit 05b9db9cf0fe8b40873b2b74e203b4fc82e7fea4
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 02:47:41 2022 +0000
3 changes:
- Removing s3 dep
- remove uesless import
- added `deployed_on` in dag file template
commit c6afba95f5ec05acf7f33fd3228cffd784556e3b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Apr 15 22:50:52 2022 +0000
Fixed passing secrets with kubernetes.
commit c3ce7e9faa5f7a23d309e2f66f778dbca85df22a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Apr 15 22:04:22 2022 +0000
Refactored code .
- removed compute/k8s.py
- Moved k8s code to airflow_compiler.py
- ran isort to airflow_compiler.py
commit d1c343dbbffbddbebd2aeda26d6846e595144e0b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Apr 15 18:02:25 2022 +0000
Added validations about:
- un-supported decorators
- foreach
Changed where validations are done to not save the package.
commit 7b19f8e66e278c75d836daf6a1c7ed2c607417ce
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Apr 15 03:34:26 2022 +0000
Fixing mf log related bug
- No double logging on metaflow.
commit 4d1f6bf9bb32868c949d8c103c8fe44ea41b3f13
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 03:10:51 2022 +0000
Removed usless code WRT project decorator.
commit 5ad9a3949e351b0ac13f11df13446953932e8ffc
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 03:03:19 2022 +0000
Remove readme.
commit 60cb6a79404efe2bcf9bf9a118a68f0b98c7d771
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 03:02:38 2022 +0000
Made file path required arguement.
commit 9f0dc1b2e01ee04b05620630f3a0ec04fe873a31
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 03:01:07 2022 +0000
changed `--is-active`->`--is-paused-upon-creation`
- dags are active by default.
commit 5b98f937a62ee74de8aed8b0efde5045a28f068b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 02:55:46 2022 +0000
shortened length of run-id and task-id hashes.
commit e53426eaa4b156e8bd70ae7510c2e7c66745d101
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 02:41:32 2022 +0000
…
npow
added a commit
to npow/metaflow
that referenced
this pull request
Jan 12, 2023
* Don't explicitly break py2 support (#962)
* Don't explicitly break py2 support
* Typo
* Typo
* Pass the paths used by the external interpreter to the one launching the Env Escape server (#960)
We were currently not reflecting possible values passed using PYTHONPATH or
programmatically added to the interpreter launching the environment escape server.
* Bump follow-redirects in /metaflow/plugins/cards/ui (#966)
Bumps [follow-redirects](https://github.com/follow-redirects/follow-redirects) from 1.14.7 to 1.14.9.
- [Release notes](https://github.com/follow-redirects/follow-redirects/releases)
- [Commits](https://github.com/follow-redirects/follow-redirects/compare/v1.14.7...v1.14.9)
---
updated-dependencies:
- dependency-name: follow-redirects
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Adding export of graph to JSON for cli (#955)
* Fix "Too many symbolic links" error when using Conda + Batch on MacOS (#972)
* Fix "Too many symbolic links" error when using Conda + Batch on MacOS
* Ran black
* emit app tag for AWS Batch jobs (#970)
* Bump to 2.5.3 (#974)
* Extension packaging improvements (#959)
* Fixed dual distribution import in some cases; fixed get_plugin_cli extension
If sys.path contains the same directory multiple times, metadata.distributions()
will list a package multiple times which breaks the import mechanism we have. This
is addressed by skipping duplicate packages.
get_plugin_cli for extensions wasn't evaluated late enough therefore
not allowing additional parameters to be added.
* Fix issue with metaflow_extensions and Conda environment
In a Conda environment, it is not possible to re-resolve all the
metaflow_extension packages since they appear as a single directory.
In that case, we resolve on the INFO file to give us the proper
information.
Also fixed the way metaflow_extensions are added to the Conda
environment to avoid leaking more information than needed.
* Validate configuration files for extensions and de-duplicate for internal ns packages
* Update packaging of metaflow_extensions packages
Packaging is now handled directly by extension_support.py and
each distribution/package can define its own suffixes to include.
* Rework the import mechanism for modules contained in metaflow_extensions
This change does away with the deprecated load_module and also improves
handling of loading children modules and _orig modules
* Address comments
* Squash merge of origin/master
* Fix issue when packages were left out
* Merge get_pinned_conda_libs
* Fix issue with __init__.py in non-distribution packages
* Moving card import changes on top of Romain's branch (#973)
* Moving card import changes on top of Romain's branch
- tiny bug fix to extensions.
- Added card related refactor to support mfextinit and regular package.
- removing card packaging related code in decorator + everywhere else.
* Tiny bug fix on top of romain's new changes.
* removing unneccessary logic
* Added a bit more inline documentation and addressed documentation comments
* Properly handle parsing of package requirements
* Fix Pylint errors in some case of metaflow_extensions module aliasing
* Properly handle case of files at the root of PYTHONPATH package
* Tests for Extensions (#978)
* Added tests for extensions. Checking if they work.
* dummy commit to see if things work.
* fix
* debug
* bug fix.
* possible fix.
* tweeking context
* Bug fix to tests.
* dummy commit
* bug fix
* Added extension test to core tests.
- remove seperate gh action
* removing files.
* fix
* added extension test to py3 context
- remove redundant complexity.
Co-authored-by: Valay Dave <valaygaurang@gmail.com>
* pass DEFAULT_AWS_CLIENT_PROVIDER to remote tasks (#982)
* Fixing some hacky plumbing in card test suite (#967)
* Fixing some hacky plumbing for card tests.
- Added `--file` arg to card list cli
- Using files to get the information written by cli
- changes to test to use files instead of stdout
- Piping stderrors to stderrr to capture card not found errors.
* Removing `\n` from all card tests assertions
- `\n` was there because earlier we read stdout.
- Now since we read files, it will not be needed.
* Simplify mflog (#979)
* Fix extension root determination in some cases + card tweaks (#989)
* Use importlib_metadata 2.8.3 for Python 3.4, 3.5 but 4.8.3 for Python 3.6+ (#988)
* Use importlib_metadata 2.8.3 for Python 3.4 and 3.5 but 4.8.3 for Python 3.6+
This is required because in importlib_metadata 3.4.0, a field called `_normalized_name`
was introduced and is relied on by later versions of importlib_metadata. When
importlib_metadata looks for distributions, it queries all registered/installed
importlib_metadata and if an older version (like 2.8.3) returns something that
does not have this field, it causes a crash.
* Add __init__.py files to make the vendored package non namespace
* Allow configuring the root directory where artifacts go when pulling from S3. (#991)
Co-authored-by: kgullikson <kgullikson@sparkcognition.com>
* Add an option to pass a `role` argument to S3() to provide a role (#987)
* Add an option to pass a `role` argument to S3() to provide a role for the S3 client
* Move to partial to be able to pickle ops
* Address comments
* bump version to 2.5.4 (#993)
* Bug fix for datetime index in default card (#981)
* Bug fix for datetime_indexes.
* ran black
* Bump minimist from 1.2.5 to 1.2.6 in /metaflow/plugins/cards/ui (#995)
Bumps [minimist](https://github.com/substack/minimist) from 1.2.5 to 1.2.6.
- [Release notes](https://github.com/substack/minimist/releases)
- [Commits](https://github.com/substack/minimist/compare/1.2.5...1.2.6)
---
updated-dependencies:
- dependency-name: minimist
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Fix black - set upper bound for click version (#1006)
* Add tags to `current` singleton (#1019)
* first attempt
* fix _set_env param typo
* add assertions for tags to current singleton test
* Dispatch Metaflow flows to Argo Workflows (#992)
* Dispatch Metaflow flows to Argo Workflows
* Remove spurious files
* Make black happy
* Make black happy again
* Remove spurious TODOs
* Handle non-numeric task ids
* Fix typos
* don't actively automount service account tokens
* make black happy
* Update @kubernetes
* remove spurious commits
* fix issue with hyphens
* raise memory for tests
* change limits to requests
* pin back test resources
* update refresh timeout
* stylistic nits
* Add lint check
* make black happy
* support gpus
* drop extra bracket
* fix
* Skip sleep if not retryting S3 operations (#1001)
* Don't sleep if not retrying S3 operations
* Remove unused env variable
* VSCode is too smart
* Bump minor version for release (#1022)
* Alternate way of adding metadata to cloned tasks (#1003)
* Alternate way of adding metadata to cloned tasks
This adds it on the client side
* Clean up metadata passing.
Also optimize use within the client
* Typo
* Fix black test
* Keep black happy
* Added tests
* Ignore namespace when getting origin task
* Bump cross-fetch from 3.1.4 to 3.1.5 in /metaflow/plugins/cards/ui (#1027)
Bumps [cross-fetch](https://github.com/lquixada/cross-fetch) from 3.1.4 to 3.1.5.
- [Release notes](https://github.com/lquixada/cross-fetch/releases)
- [Commits](https://github.com/lquixada/cross-fetch/compare/v3.1.4...v3.1.5)
---
updated-dependencies:
- dependency-name: cross-fetch
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fixed bug from #1023 (#1025)
* current.pathspec to return None when used outside Flow (#1033)
* current.pathspec to return None when used outside Flow
* formatted with black
* Fixing bug in the `card list` command (#1044)
- echo_always defaulted to stderr :(
* Enable MinIO/EMC-ECS as blob stores for Metaflow (#1045)
* Enable EMCECS
* Enable MinIO & EMC-ECS blob stores for Metaflow
* fix InvalidRange issue
* [tag-mutation-project] Stash system info in task metadata (#1039)
* stash system info as metadata (in addition to system_tags today)
* improve
* avoid parsing on ":" for user id
* remove date - it's not well defined and created_at is available
* python_version is in metadata now, and can vary between orig and resume runs
* address CR comments
* add the comments
* Address issues with S3 get and ranges (#1034)
* Address issues with S3 get and ranges
This addresses two issues:
- get and get_many would not properly populate the range_info in S3Object
- get_many on different ranges of the same file would return incorrect results
* Add one more test (merge with Netflix internal tests)
* Put Run() test behind a check
* Address comments
* Fix _new_task calling bug in LocalMetadataProvider (#1046)
* Release 2.6.1 (#1047)
* Support default secrets (#1048)
* Run id file should be written prior to execution when resuming (#1051)
* Run id file should be written prior to execution when resuming
* separate run id test so we can exclude from batch
* release 2.6.2 (#1052)
* Fix instance metadata calls for IMDSV2 (#1053)
* bump version to 2.6.3 (#1054)
* Small reorder of imports to make things more consistent (#1055)
* [tag-mutation-project] Local metadata provider - object read paths to return ancestral Run tags (#1043)
* wip
* Fix nits
* typo fix. (#1068)
* This is the Jackie's tag changes with an additional test and a fix. (#1063)
* rebase on master
* add replace_many as candidate subcommand
* updates from some CR suggestions
* propagate opt rename
* more changes follow CR
* improve consistency guarantees of local metadata tag mutations (retries if a race was lost)
* write comment
* use replace-many logic to back "tag replace"
* finalize client API for tag operations
* fix retry logic
* Add task.tags == run.tags check in test
* [tag-mutation-project] Add tag mutation support to ServiceMetadataProvider
* fix version
* Enable TagMutationTest in batch and k8s
* Bogus commit
* minor UX fixes on CLI
* Fully support CliCheck for TagMutationTest
* add tag and tag set validation to local metadata code path
Validate tags on run / resume paths too
* flat nor flatten on CLI
* Logs really do come via stdout - related to CliCheck refactor
* argo and step functions - validate tags as soon as possible (before other user messaging)
* no capture_output pre 3.7
* Add another test for tag mutation
* Properly pass down the metadata provider
There were two cases when the metadata provider was not properly
passed down to the client:
- when calling the client within a flow
- when invoking the `tag` command
This fixes both issues and also allows the user to introspect
the metadata provider and datastore being used for the flow
* Do not make public metadata and datastore in current object
* Typo
Co-authored-by: Jackie Tung <jackie@outerbounds.co>
Co-authored-by: jackie-ob <104396218+jackie-ob@users.noreply.github.com>
* Add two options to resume: the ability to specify a run-id and the ab… (#1059)
* Add two options to resume: the ability to specify a run-id and the ability to only clone tasks
The scenario for this use is mainly for external schedulers that are
capable of "resuming" failed runs by only re-executing the ones that failed
or didn't execute the first time. We need a way for Metaflow to "clone" the
tasks that are not re-executed.
* Small formatting fix
* Made clone-only resume possibly re-entrant
* Addressed comments
* Clarify the reentrant behavior of resume
* Improve sidecar message handling (#1057)
* Improve sidecar message handling
Several changes:
- allow side-passing of a static "context" to the sidecar process to keep
messages as small as possible
- clean up the interfaces and make it clearer what is what (ie: there is
an emitter inside the process and then a sidecar that runs outside)
- better error checking
- better handling of shutdown (sidecars will now properly get a shutdown
message giving them the opportunity to clean up -- this may add a bit
of latency at the end of the program execution.
Also improved message when plugins don't load in case of an issue with
Metaflow extensions
* Add the possibility to pass additional context to sidecars
This allows more information to be provided after the sidecar is
created but still maintains small message sizes
* Better handling of initial context message for sidecars
In the previous implementation, a failure to send the initial context
message could crash the flow. This change properly handles that message
like any other message (so has no impact on the flow itself). It also
implements better retry policies to send the context. Finally, this change
also improves error handling: if a message fails to send, in the previous
implementation, it was likely that the next successful one would be
an invalid json.
* Improve handling of corner conditions on message send
* Restart sidecar faster when sidecar process dies
* Properly send context after sidecar restart
* Remove the specific context from monitors/loggers
Subclasses can now handle contexts directly in the subclass.
Refactoring also cleaned up what a sidecar and a sidecar worker
do by refactoring some of the code out.
* Typo: fix names
* Minor typos for sidecar changes
* Clean up sidecars
* More cleanups for sidecars
* Addressed final comments for sidecars
* Small typo in NullEventLogger.send (#1071)
Pushing directly per our conversation with Savin
* Bump version to 2.7.0
* Change behavior of env escape when module is present (#1072)
When a module configured in the env escape was present in the local
environment, we would raise an error. We now print a message and use
the module present in the environment (not the environment escape).
This allows the env escape modules to provide a "fallback" in case the
module is not present but to still use the local module if present
* Bump version to 2.7.1 (#1073)
* Fix an issue with the environment escape server directory (#1074)
The environment escape server would launch in the cwd. This could
cause issues if, for example, a `metaflow` directory existed in that
cwd (because it would then take precedence over the actual metaflow.
To remediate this situation, the server is launched in the configuration
directory (where we know what there is).
Also took care of a quick annoyance of having to create an empty overrides.py
file for everything. This is now no longer required
* Support session_vars and client_params in S3() (#1069)
* Support session_vars and client_params in S3()
This allows the user to pass down session variables and client parameters
to set when getting a S3 client.
* Addressed comments about default configuration
* Support M1 Macs (#1077)
* changes to @kubernetes for sandbox (#1079)
* Changed error to warning in AWS retry (#1081)
Co-authored-by: Preetam Joshi <preetamj@netflix.com>
* bump to 2.7.2 (#1083)
* Fixed s3util test (#1084)
* Fixing s3util test caset
* added comment
Co-authored-by: Preetam Joshi <preetamj@netflix.com>
* Bump svelte from 3.46.1 to 3.49.0 in /metaflow/plugins/cards/ui (#1086)
Bumps [svelte](https://github.com/sveltejs/svelte) from 3.46.1 to 3.49.0.
- [Release notes](https://github.com/sveltejs/svelte/releases)
- [Changelog](https://github.com/sveltejs/svelte/blob/master/CHANGELOG.md)
- [Commits](https://github.com/sveltejs/svelte/compare/v3.46.1...v3.49.0)
---
updated-dependencies:
- dependency-name: svelte
dependency-type: direct:development
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Metadata version check flag (#1088)
* Bump terser from 5.10.0 to 5.14.2 in /metaflow/plugins/cards/ui (#1090)
Bumps [terser](https://github.com/terser/terser) from 5.10.0 to 5.14.2.
- [Release notes](https://github.com/terser/terser/releases)
- [Changelog](https://github.com/terser/terser/blob/master/CHANGELOG.md)
- [Commits](https://github.com/terser/terser/commits)
---
updated-dependencies:
- dependency-name: terser
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* fix r CI deps (#1092)
* fix r CI deps
* fix fractional resoure handling for batch (#1089)
* bump version (#1093)
* Fix docstrings for the API reference (no functional changes!) (#1076)
Make the docstring format in public APIs compliant with the new API reference framework
* Move a sys.path modification in s3op to __main__ (#1095)
In its current location, this could modify the sys.path of the current
running metaflow which could have nefarious consequences with the escape
hatch which uses sys.path to determine the outside environment's python path.
The following scenario would cause issues:
- metaflow is installed in the usual path on the system
- a conda environment was manually bootstrapped from a directory A.
+ at this point, sys.path starts with `A` and then contains the other
system includes
+ s3op.py is imported at some point by the Conda installer when it calls `get_many`
+ this modifies sys.path to insert, at the beginning, the parent of Metaflow; so
in this case, sys.path looks something like
['/apps/python3/lib/python3.7/site-packages', 'A', '/apps/python3/lib/python3.7/site-packages'...]
+ when the escape hatch trampolines are created, this sys.path is used to determine what the
sys.path for the outside interpreter is.
+ in A, we create:
* INFO
* metaflow
* metaflow_extensions
which properly describe the installation of metaflow
- when the escape hatch client runs, it runs in the conda environment and uses metaflow
created in A.
- when the client wants to start the server, this is where we run into issues because, at this
point, the server will use the PYTHONPATH which starts with '/apps/python3/lib/python3.7/site-packages'
in which it will find metaflow. It will therefore use that metaflow (which is the same
as the one linked in A) to start the server. This runs into issues though because A
is also in PYTHONPATH and so the extension support loader will also try to load `A/metaflow_extensions`.
This will cause issues if multiple extensions are installed there (it will complain about duplicate
configurations for example. The `INFO` file typically used to solve this problem is not read as it was
not present for the TL metaflow.
This patch simply moves the modification of sys.path to where it is actually needed and avoids polluting
sys.path when the module is simply included (and not called as a script).
* Airflow Support (#1094)
* Airflow on Kubernetes minus Foreachs.
- Support for all metaflow construct without foreach and sensors
Squashed commit of the following:
commit ef8b1e3768695bc4d3375a947ab1da9c6520bcf1
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jul 29 01:06:26 2022 +0000
Removed sernsors and banned foreach's
commit 8d517c4fecc6568777ad03eca81aaacfa3e91156
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jul 29 00:59:01 2022 +0000
commiting k8s related file from master.
commit a7e1ecdbf7b8b8d1cc21321cc8e196053f8305e4
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jul 29 00:54:45 2022 +0000
Uncommented code for foreach support with k8s
KubernetesPodOperator version 4.2.0 renamed `resources` to
`container_resources`
- Check : (https://github.com/apache/airflow/pull/24673) /
- (https://github.com/apache/airflow/commit/45f4290712f5f779e57034f81dbaab5d77d5de85)
This was done because `KubernetesPodOperator` didn't play nice with dynamic task mapping and they had to deprecate the `resources` argument. Hence the below codepath checks for the version of `KubernetesPodOperator`
and then sets the argument. If the version < 4.2.0 then we set the argument as `resources`.
If it is > 4.2.0 then we set the argument as `container_resources`
The `resources` argument of KuberentesPodOperator is going to be deprecated soon in the future.
So we will only use it for `KuberentesPodOperator` version < 4.2.0
The `resources` argument will also not work for foreach's.
commit 2719f5d792ada91e3ae0af6f1a9a0c7d90f74660
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:31:58 2022 +0000
nit fixes :
- fixing comments.
- refactor some variable/function names.
commit 2079293fbba0d3d862476a7d67b36af8a3389342
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:14:53 2022 +0000
change `token` to `production_token`
commit 14aad5ff717418e4183a88fa84b2f5e5bb13927a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:11:56 2022 +0000
Refactored import Airflow Sensors.
commit b1472d5f7a629024ca45e8b83700400d02a4d455
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:08:41 2022 +0000
new comment on `startup_timeout_seconds` env var.
commit 6d81b758e8f06911258d26f790029f557488a0d7
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:06:09 2022 +0000
Removing traces of `@airflow_schedule_interval`
commit 0673db7475b22f3ce17c2680fc0a7c4271b5c946
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Jul 14 12:43:08 2022 -0700
Foreach polish (valayDave/metaflow#62)
* Removing unused imports
* Added validation logic for airflow version numbers with foreaches
* Removed `airflow_schedule_interval` decorator.
* Added production/deployment token related changes
- Uses s3 as a backend to store the production token
- Token used for avoiding nameclashes
- token stored via `FlowDatastore`
* Graph type validation for airflow foreachs
- Airflow foreachs only support single node fanout.
- validation invalidates graphs with nested foreachs
* Added configuration about startup_timeout.
* Added final todo on `resources` argument of k8sOp
- added a commented code block
- it needs to be uncommented when airflow releasese the patch for the op
- Code seems feature complete keeping aside airflow patch
commit 4b2dd1211fe2daeb76e29e4084f21e96b10cdae9
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Jul 7 19:33:07 2022 +0000
Removed retries from user-defaults.
commit 0e87a97fea15ba3aaa6d4228b141bd796b767c43
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Jul 6 16:29:33 2022 +0000
updated pod startup time
commit fce2bd263f368dbb78a34ac71f64e13c89277222
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Jun 29 18:44:11 2022 +0000
Adding default 1 retry for any airflow worker.
commit 5ef6bbcde51b1f4923a192291ed0e07d07ec7321
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 27 01:22:42 2022 +0000
Airflow Foreach Integration
- Simple one node foreach-join support as gaurenteed by airflow
- Fixed env variable setting issue
- introduced MetaflowKuberentesOperator
- Created a new operator to allow smootness in plumbing xcom values
- Some todos
commit d319fa915c558d82f1d127736ce34d3ae0da521d
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 24 21:12:09 2022 +0000
simplifying run-id macro.
commit 0ffc813b1c4e6ba0103be51520f42d191371741a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 24 11:51:42 2022 -0700
Refactored parameter macro settings. (valayDave/metaflow#60)
commit a3a495077f34183d706c0edbe56d6213766bf5f6
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 24 02:05:57 2022 +0000
added comment on need for `start_date`
commit a3147bee08a260aa78ab2fb14c6232bfab2c2dec
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Jun 21 06:03:56 2022 +0000
Refactored an `id_creator` method.
commit 04d7f207ef2dae0ce2da2ec37163ac871f4517bc
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Jun 21 05:52:05 2022 +0000
refactor :
-`RUN_ID_LEN` to `RUN_HASH_ID_LEN`
- `TASK_ID_LEN` to `TASK_ID_HASH_LEN`
commit cde4605cd57ad9214f5a6afd7f58fe4c377e09e2
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Jun 21 05:48:55 2022 +0000
refactored an error string
commit 11458188b6c59d044fca0dd2d1f5024ec84f6488
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 20 22:42:36 2022 -0700
addressing savins comments. (#59)
- Added many adhoc changes based for some comments.
- Integrated secrets and `KUBERNETES_SECRETS`
- cleaned up parameter setting
- cleaned up setting of scheduling interval
- renamed `AIRFLOW_TASK_ID_TEMPLATE_VALUE` to `AIRFLOW_TASK_ID`
- renamed `AirflowSensorDecorator.compile` to `AirflowSensorDecorator.validate`
- Checking if dagfile and flow file are same.
- fixing variable names.
- checking out `kubernetes_decorator.py` from master (6441ed5)
- bug fixing secret setting in airflow.
- simplified parameter type parsing logic
- refactoring airflow argument parsing code.
commit 83b20a7c6a13b3aedb7e603e139f07f0ef2fb646
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 13 14:02:57 2022 -0700
Addressing Final comments. (#57)
- Added dag-run timeout.
- airflow related scheduling checks in decorator.
- Auto naming sensors if no name is provided
- Annotations to k8s operators
- fix: argument serialization for `DAG` arguments (method names refactored like `to_dict` became `serialize`)
- annotation bug fix
- setting`workflow-timeout` for only scheduled dags
commit 4931f9c84e6a1d20fc3ecb41cf138b72e5dee629
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:50:49 2022 +0000
k8s bug fix
commit 200ae8ed4a00028f094281f73a939e7a4dcdf83a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:39:50 2022 +0000
removed un-used function
commit 70e285e9a7cfbec71fc293508a62c96f33562a01
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:38:37 2022 +0000
Removed unused `sanitize_label` function
commit 84fc622d8b11e718a849b2e2d91ceb3ea69917e6
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:37:34 2022 +0000
GPU support added + container naming same as argo
commit c92280d8796ec12b4ff17fa2ff3c736c7244f39c
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:25:17 2022 +0000
Refactored sensors to different files + bug fix
- bug caused due `util.compress_list`.
- The function doesn't play nice with strings with variety of characters.
- Ensured that exceptions are handled appropriately.
- Made new file for each sensor under `airflow.sensors` module.
commit b72a1dcf0dbbbcb814581d92738fd27ec31ef673
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Jun 4 01:41:49 2022 +0000
ran black.
commit 558c82f65b383ed0d61ded6bc80326471e284550
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 18:32:48 2022 -0700
Moving information from airflow_utils to compiler (#56)
- commenting todos to organize unfinished changes.
- some environment variables set via`V1EnvVar`
- `client.V1ObjectFieldSelector` mapped env vars were not working in json form
- Moving k8s operator import into its own function.
- env vars moved.
commit 9bb5f638792a671164ec95891e97f599e9a3385f
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 18:06:03 2022 +0000
added mising Run-id prefixes to variables.
- merged `hash` and `dash_connect` filters.
commit 37b5e6a9d8ca93cc91244c8d77c7d4f61280ba59
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 18:00:22 2022 +0000
nit fix : variable name change.
commit 660756f952ebd92ba1e26d7f908b81036c31ff10
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 17:58:34 2022 +0000
nit fixes to dag.py's templating variables.
commit 1202f5bc92f76df52b5957f11c8574cadfa62196
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 17:56:53 2022 +0000
Fixed defaults passing
- Addressed comments for airflow.py
commit b9387dd428c1a37f9a3bfe2c72cab475da708c02
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 17:52:24 2022 +0000
Following Changes:
- Refactors setting scheduling interval
- refactor dag file creating function
- refactored is_active to is_paused_upon_creation
- removed catchup
commit 054e3f389febc6c447494a1dedb01228f5f5650f
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 17:33:25 2022 +0000
Multiple Changes based on comments:
1. refactored `create_k8s_args` into _to_job
2. Addressed comments for snake casing
3. refactored `attrs` for simplicity.
4. refactored `metaflow_parameters` to `parameters`.
5. Refactored setting of `input_paths`
commit d481b2fca7914b6b657a69af407cfe1a894a46dc
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 16:42:24 2022 +0000
Removed Sensor metadata extraction.
commit d8e6ec044ef8c285d7fbe1b83c10c07d51c063e3
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 16:30:34 2022 +0000
porting savin's comments
- next changes : addressing comments.
commit 3f2353a647e53bc240e28792769c42a71ea8f8c9
Merge: d370ffb c1ff469
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Jul 28 23:52:16 2022 +0000
Merge branch 'master' into airflow
commit d370ffb248411ad4675f9d55de709dbd75d3806e
Merge: a82f144 e4eb751
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Jul 14 19:38:48 2022 +0000
Merge branch 'master' into airflow
commit a82f1447b414171fc5611758cb6c12fc692f55f9
Merge: bdb1f0d 6f097e3
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Jul 13 00:35:49 2022 +0000
Merge branch 'master' into airflow
commit bdb1f0dd248d01318d4a493c75b6f54248c7be64
Merge: 8511215 f9a4968
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Jun 29 18:44:51 2022 +0000
Merge branch 'master' into airflow
commit 85112158cd352cb7de95a2262c011c6f43d98283
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Jun 21 02:53:11 2022 +0000
Bug fix from master merge.
commit 90c06f12bb14eda51c6a641766c5f67d6763abaa
Merge: 0fb73af 6441ed5
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 20 21:20:20 2022 +0000
Merge branch 'master' into airflow
commit 0fb73af8af9fca2875261e3bdd305a0daab1b229
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Jun 4 00:53:10 2022 +0000
squashing bugs after changes from master.
commit 09c6ba779f6b1b6ef1d7ed5b1bb2be70ec76575d
Merge: 7bdf662 ffff49b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Jun 4 00:20:38 2022 +0000
Merge branch 'master' into af-mmr
commit 7bdf662e14966b929b8369c65d5bd3bbe5741937
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon May 16 17:42:38 2022 -0700
Airflow sensor api (#3)
* Fixed run-id setting
- Change gaurentees that multiple dags triggered at same moment have unique run-id
* added allow multiple in `Decorator` class
* Airflow sensor integration.
>> support added for :
- ExternalTaskSensor
- S3KeySensor
- SqlSensor
>> sensors allow multiple decorators
>> sensors accept those arguments which are supported by airflow
* Added `@airflow_schedule_interval` decorator
* Fixing bug run-id related in env variable setting.
commit 2604a29452e794354cf4c612f48bae7cf45856ee
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 21 18:26:59 2022 +0000
Addressed comments.
commit 584e88b679fed7d6eec8ce564bf3707359170568
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:33:55 2022 +0000
fixed printing bug
commit 169ac1535e5567149d94749ddaf70264e882d62c
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:30:59 2022 +0000
Option help bug fix.
commit 6f8489bcc3bd715b65d8a8554a0f3932dc78c6f5
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:25:54 2022 +0000
variable renamemetaflow_specific_args
commit 0c779abcd1d9574878da6de8183461b53e0da366
Merge: d299b13 5a61508
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:23:10 2022 +0000
Merge branch 'airflow-tests' into airflow
commit 5a61508e61583b567ef8d3fea04e049d74a6d973
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:22:54 2022 +0000
Removing un-used code / resolved-todos.
commit d030830f2543f489a1c4ebd17da1b47942f041d6
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:06:03 2022 +0000
ran black,
commit 2d1fc06e41cbe45ccfd46e03bc87b09c7a78da45
Merge: f2cb319 7921d13
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:04:19 2022 +0000
Merge branch 'master' into airflow-tests
commit d299b13ce38d027ab27ce23c9bbcc0f43b222cfa
Merge: f2cb319 7921d13
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:02:37 2022 +0000
Merge branch 'master' into airflow
commit f2cb3197725f11520da0d49cbeef8de215c243eb
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 02:54:03 2022 +0000
reverting change.
commit 05b9db9cf0fe8b40873b2b74e203b4fc82e7fea4
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 02:47:41 2022 +0000
3 changes:
- Removing s3 dep
- remove uesless import
- added `deployed_on` in dag file template
commit c6afba95f5ec05acf7f33fd3228cffd784556e3b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Apr 15 22:50:52 2022 +0000
Fixed passing secrets with kubernetes.
commit c3ce7e9faa5f7a23d309e2f66f778dbca85df22a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Apr 15 22:04:22 2022 +0000
Refactored code .
- removed compute/k8s.py
- Moved k8s code to airflow_compiler.py
- ran isort to airflow_compiler.py
commit d1c343dbbffbddbebd2aeda26d6846e595144e0b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Apr 15 18:02:25 2022 +0000
Added validations about:
- un-supported decorators
- foreach
Changed where validations are done to not save the package.
commit 7b19f8e66e278c75d836daf6a1c7ed2c607417ce
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Apr 15 03:34:26 2022 +0000
Fixing mf log related bug
- No double logging on metaflow.
commit 4d1f6bf9bb32868c949d8c103c8fe44ea41b3f13
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 03:10:51 2022 +0000
Removed usless code WRT project decorator.
commit 5ad9a3949e351b0ac13f11df13446953932e8ffc
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 03:03:19 2022 +0000
Remove readme.
commit 60cb6a79404efe2bcf9bf9a118a68f0b98c7d771
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 03:02:38 2022 +0000
Made file path required arguement.
commit 9f0dc1b2e01ee04b05620630f3a0ec04fe873a31
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 03:01:07 2022 +0000
changed `--is-active`->`--is-paused-upon-creation`
- dags are active by default.
commit 5b98f937a62ee74de8aed8b0efde5045a28f068b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 02:55:46 2022 +0000
shortened length of run-id and task-id hashes.
commit e53426eaa4b156e8bd70ae7510c2e7c66745d101
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 02:41:32 2022 +0000
Removing un-used args.
commit 72cbbfc7424f9be415c22d9144b16a0953f15295
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 02:39:59 2022 +0000
Moved exceptions to airflow compiler
commit b2970ddaa86c393c8abb7f203f6507c386ecbe00
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 02:33:02 2022 +0000
Changes based on PR comments:
- removed airflow xcom push file , moved to decorator code
- removed prefix configuration
- nit fixes.
commit 9e622bac5a75eb9e7a6594d8fa0e47f076634b44
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Apr 11 20:39:00 2022 +0000
Removing un-used code paths + code cleanup
commit 7425f62cff2c9128eea785223ddeb40fa2d8f503
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Apr 11 19:45:04 2022 +0000
Fixing bug fix in schedule.
commit eb775cbadd1d2d2c90f160a95a0f42c8ff0d7f4c
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Apr 10 02:52:59 2022 +0000
Bug fixes WRT Kubernetes secrets + k8s deployments.
- Fixing some error messages.
- Added some comments.
commit 04c92b92c312a4789d3c1e156f61ef57b08dba9f
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Apr 10 01:20:53 2022 +0000
Added secrets support.
commit 4a0a85dff77327640233767e567aee2b379ac13e
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Apr 10 00:11:46 2022 +0000
Bug fix.
commit af91099c0a30c26b58d58696a3ef697ec49a8503
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Apr 10 00:03:34 2022 +0000
bug fix.
commit c17f04a253dfe6118e2779db79da9669aa2fcef2
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Apr 9 23:55:41 2022 +0000
Bug fix in active defaults.
commit 0d372361297857076df6af235d1de7005ac1544a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Apr 9 23:54:02 2022 +0000
@project, @schedule, default active dag support.
- Added a flag to allow setting dag as active on creation
- Airflow compatible schedule interval
- Project name fixes.
commit 5c97b15cb11b5e8279befc5b14c239463750e9b7
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 7 21:15:18 2022 +0000
Max workers and worker pool support.
commit 9c973f2f44c3cb3a98e3e63f6e4dcef898bc8bf2
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 7 19:34:33 2022 +0000
Adding exceptions for missing features.
commit 2a946e2f083a34b4b6ed84c70aebf96b084ee8a2
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Mar 28 19:34:11 2022 +0000
2 changes :
- removed hacky line
- added support to directly throw dags in s3.
commit e0772ec1bad473482c6fd19f8c5e8b9845303c0a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Mar 23 22:38:20 2022 +0000
fixing bugs in service account setting
commit 874b94aeeabc664f12551864eff9d8fdc24dc37b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Mar 20 23:49:15 2022 +0000
Added support for Branching with Airflow
- remove `next` function in `AirflowTask`
- `AirflowTask`s has no knowledge of next tasks.
- removed todos + added some todos
- Graph construction on airflow side using graph_structure datastructure.
- graph_structure comes from`FlowGraph.output_steps()[1]`
commit 8e9f649bd8c51171c38a1e5af70a44a85e7009ca
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Mar 20 02:33:04 2022 +0000
Added hacky line
commit fd5db04cf0a81b14efda5eaf40cd9227e2bac0d3
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Mar 20 02:06:38 2022 +0000
Removed hacky line.
commit 5b23eb7d8446bef71246d853b11edafa93c6ef95
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Mar 20 01:44:57 2022 +0000
Added support for Parameters.
- Supporting int, str, bool, float, JSONType
commit c9378e9b284657357ad2997f2b492bc2f4aaefac
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sun Mar 20 00:14:10 2022 +0000
Removed todos + added some validation logic.
commit 7250a44e1dea1da3464f6f71d0c5188bd314275a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Mar 19 23:45:15 2022 +0000
Fixing logs related change from master.
commit d125978619ab666dcf96db330acdca40f41b7114
Merge: 8cdac53 7e210a2
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Mar 19 23:42:48 2022 +0000
Merge branch 'master' into aft-mm
commit 8cdac53dd32648455e36955badb8e0ef7b95a2b3
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Mar 19 23:36:47 2022 +0000
making changes sync with master
commit 5a93d9f5198c360b2a84ab13a86496986850953c
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Mar 19 23:29:47 2022 +0000
Fixed bug when using catch + retry
commit 62bc8dff68a6171b3b4222075a8e8ac109f65b4c
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Mar 19 22:58:37 2022 +0000
Changed retry setting.
commit 563a20036a2dfcc48101f680f29d4917d53aa247
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Mar 19 22:42:57 2022 +0000
Fixed setting `task_id` :
- switch task-id from airflow job is to hash to "runid/stepname"
- refactor xcom setting variables
- added comments
commit e2a1e502221dc603385263c82e2c068b9f055188
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Mar 19 17:51:59 2022 +0000
setting retry logic.
commit a697b56052210c8f009b68772c902bbf77713202
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Mar 17 01:02:11 2022 +0000
Nit fix.
commit 68f13beb17c7e73c0dddc142ef2418675a506439
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Mar 16 20:46:19 2022 +0000
Added @schedule support + readme
commit 57bdde54f9ad2c8fe5513dbdb9fd02394664e234
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Mar 15 19:47:06 2022 +0000
Fixed setting run-id / task-id to labels in k8s
- Fixed setting run-id has from cli macro
- added hashing macro to ensure that jinja template set the correct run-id to k8s labels
-
commit 3d6c31917297d0be5f9915b13680fc415ddb4421
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Mar 15 05:39:04 2022 +0000
Got linear workflows working on airflow.
- Still not feature complete as lots of args are still unfilled / lots of unknows
- minor tweek in eks to ensure airflow is k8s compatible.
- passing state around via xcom-push
- HACK : AWS keys are passed in a shady way. : Reverse this soon.
commit db074b8012f76d9d85225a4ceddb2cde8fefa0f4
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Mar 11 12:34:33 2022 -0800
Tweeks
commit a9f0468c4721a2017f1b26eb8edcdd80aaa57203
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Mar 1 17:14:47 2022 -0800
some changes based on savin's comments.
- Added changes to task datastore for different reason : (todo) Decouple these
- Added comments to SFN for reference.
- Airflow DAG is no longer dependent on metaflow
commit f32d089cd3865927bc7510f24ba3418d859410b6
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Feb 23 00:54:17 2022 -0800
First version of dynamic dag compiler.
- Not completely finished code
- Creates generic .py file a JSON that is parsed to create Airflow DAG.
- Currently only boiler plate to make a linear dag but doesn't execute anything.
- Unfinished code.
commit d2def665a86d6a6622d6076882c1c2d54044e773
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Feb 19 14:01:47 2022 -0800
more tweeks.
commit b176311f166788cc3dfc93354a0c5045a4e6a3d4
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Feb 17 09:04:29 2022 -0800
commit 0
- unfinished code.
* Making version compatibility changes.
- Minimum support version to 2.2.0
* Task-id macro related logic refactor
- Done for better version support.
* bug fix: param related task-id setting.
* applied black
* Reverting `decorators.py` to master
* Move sys.path insert earlier in s3op.py (#1098)
* Update setup.py (#1099)
* Fix for env_escape bug when importing local packages (#1100)
* Bump to 2.7.5 (#1102)
* Fix another issue with the escape hatch and paths (#1105)
* Bump to 2.7.6 (#1106)
* add a flag to overwrite config when running metaflow configure sandbox (#1103)
* card dev docs tiny fix. (#1108)
* Fix an issue with get_cards not respecting a Task's ds-root (#1111)
get_cards would not always respect a Task's ds-root leading to cases
where a Task has cards but they cannot be accessed because of an invalid
path.
* Adding support for Azure Blob Storage as a datastore (#1091)
* Azure Storage
* fix BrokenProcessPool import for older pythons
* Fix batch
* add azure configure, cannot validate storage account url after all (catch-22 in configure)
* rename storage account url to AZURE_STORAGE_BLOB_SERVICE_ENDPOINT
* fix indent
* clean up kubernetes config cli, add secrets
* remove signature
* fix fractional resoure handling for batch (#1089)
* bump version (#1093)
* Fix docstrings for the API reference (no functional changes!) (#1076)
Make the docstring format in public APIs compliant with the new API reference framework
* Fix issue with get_pinned_conda_libs and metaflow extensions
* clean up includefile a bit, remove from_env (not used)
* Move a sys.path modification in s3op to __main__ (#1095)
In its current location, this could modify the sys.path of the current
running metaflow which could have nefarious consequences with the escape
hatch which uses sys.path to determine the outside environment's python path.
The following scenario would cause issues:
- metaflow is installed in the usual path on the system
- a conda environment was manually bootstrapped from a directory A.
+ at this point, sys.path starts with `A` and then contains the other
system includes
+ s3op.py is imported at some point by the Conda installer when it calls `get_many`
+ this modifies sys.path to insert, at the beginning, the parent of Metaflow; so
in this case, sys.path looks something like
['/apps/python3/lib/python3.7/site-packages', 'A', '/apps/python3/lib/python3.7/site-packages'...]
+ when the escape hatch trampolines are created, this sys.path is used to determine what the
sys.path for the outside interpreter is.
+ in A, we create:
* INFO
* metaflow
* metaflow_extensions
which properly describe the installation of metaflow
- when the escape hatch client runs, it runs in the conda environment and uses metaflow
created in A.
- when the client wants to start the server, this is where we run into issues because, at this
point, the server will use the PYTHONPATH which starts with '/apps/python3/lib/python3.7/site-packages'
in which it will find metaflow. It will therefore use that metaflow (which is the same
as the one linked in A) to start the server. This runs into issues though because A
is also in PYTHONPATH and so the extension support loader will also try to load `A/metaflow_extensions`.
This will cause issues if multiple extensions are installed there (it will complain about duplicate
configurations for example. The `INFO` file typically used to solve this problem is not read as it was
not present for the TL metaflow.
This patch simply moves the modification of sys.path to where it is actually needed and avoids polluting
sys.path when the module is simply included (and not called as a script).
* Airflow Support (#1094)
* Airflow on Kubernetes minus Foreachs.
- Support for all metaflow construct without foreach and sensors
Squashed commit of the following:
commit ef8b1e3768695bc4d3375a947ab1da9c6520bcf1
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jul 29 01:06:26 2022 +0000
Removed sernsors and banned foreach's
commit 8d517c4fecc6568777ad03eca81aaacfa3e91156
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jul 29 00:59:01 2022 +0000
commiting k8s related file from master.
commit a7e1ecdbf7b8b8d1cc21321cc8e196053f8305e4
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jul 29 00:54:45 2022 +0000
Uncommented code for foreach support with k8s
KubernetesPodOperator version 4.2.0 renamed `resources` to
`container_resources`
- Check : (https://github.com/apache/airflow/pull/24673) /
- (https://github.com/apache/airflow/commit/45f4290712f5f779e57034f81dbaab5d77d5de85)
This was done because `KubernetesPodOperator` didn't play nice with dynamic task mapping and they had to deprecate the `resources` argument. Hence the below codepath checks for the version of `KubernetesPodOperator`
and then sets the argument. If the version < 4.2.0 then we set the argument as `resources`.
If it is > 4.2.0 then we set the argument as `container_resources`
The `resources` argument of KuberentesPodOperator is going to be deprecated soon in the future.
So we will only use it for `KuberentesPodOperator` version < 4.2.0
The `resources` argument will also not work for foreach's.
commit 2719f5d792ada91e3ae0af6f1a9a0c7d90f74660
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:31:58 2022 +0000
nit fixes :
- fixing comments.
- refactor some variable/function names.
commit 2079293fbba0d3d862476a7d67b36af8a3389342
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:14:53 2022 +0000
change `token` to `production_token`
commit 14aad5ff717418e4183a88fa84b2f5e5bb13927a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:11:56 2022 +0000
Refactored import Airflow Sensors.
commit b1472d5f7a629024ca45e8b83700400d02a4d455
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:08:41 2022 +0000
new comment on `startup_timeout_seconds` env var.
commit 6d81b758e8f06911258d26f790029f557488a0d7
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jul 18 18:06:09 2022 +0000
Removing traces of `@airflow_schedule_interval`
commit 0673db7475b22f3ce17c2680fc0a7c4271b5c946
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Jul 14 12:43:08 2022 -0700
Foreach polish (valayDave/metaflow#62)
* Removing unused imports
* Added validation logic for airflow version numbers with foreaches
* Removed `airflow_schedule_interval` decorator.
* Added production/deployment token related changes
- Uses s3 as a backend to store the production token
- Token used for avoiding nameclashes
- token stored via `FlowDatastore`
* Graph type validation for airflow foreachs
- Airflow foreachs only support single node fanout.
- validation invalidates graphs with nested foreachs
* Added configuration about startup_timeout.
* Added final todo on `resources` argument of k8sOp
- added a commented code block
- it needs to be uncommented when airflow releasese the patch for the op
- Code seems feature complete keeping aside airflow patch
commit 4b2dd1211fe2daeb76e29e4084f21e96b10cdae9
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Jul 7 19:33:07 2022 +0000
Removed retries from user-defaults.
commit 0e87a97fea15ba3aaa6d4228b141bd796b767c43
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Jul 6 16:29:33 2022 +0000
updated pod startup time
commit fce2bd263f368dbb78a34ac71f64e13c89277222
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Jun 29 18:44:11 2022 +0000
Adding default 1 retry for any airflow worker.
commit 5ef6bbcde51b1f4923a192291ed0e07d07ec7321
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 27 01:22:42 2022 +0000
Airflow Foreach Integration
- Simple one node foreach-join support as gaurenteed by airflow
- Fixed env variable setting issue
- introduced MetaflowKuberentesOperator
- Created a new operator to allow smootness in plumbing xcom values
- Some todos
commit d319fa915c558d82f1d127736ce34d3ae0da521d
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 24 21:12:09 2022 +0000
simplifying run-id macro.
commit 0ffc813b1c4e6ba0103be51520f42d191371741a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 24 11:51:42 2022 -0700
Refactored parameter macro settings. (valayDave/metaflow#60)
commit a3a495077f34183d706c0edbe56d6213766bf5f6
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 24 02:05:57 2022 +0000
added comment on need for `start_date`
commit a3147bee08a260aa78ab2fb14c6232bfab2c2dec
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Jun 21 06:03:56 2022 +0000
Refactored an `id_creator` method.
commit 04d7f207ef2dae0ce2da2ec37163ac871f4517bc
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Jun 21 05:52:05 2022 +0000
refactor :
-`RUN_ID_LEN` to `RUN_HASH_ID_LEN`
- `TASK_ID_LEN` to `TASK_ID_HASH_LEN`
commit cde4605cd57ad9214f5a6afd7f58fe4c377e09e2
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Jun 21 05:48:55 2022 +0000
refactored an error string
commit 11458188b6c59d044fca0dd2d1f5024ec84f6488
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 20 22:42:36 2022 -0700
addressing savins comments. (#59)
- Added many adhoc changes based for some comments.
- Integrated secrets and `KUBERNETES_SECRETS`
- cleaned up parameter setting
- cleaned up setting of scheduling interval
- renamed `AIRFLOW_TASK_ID_TEMPLATE_VALUE` to `AIRFLOW_TASK_ID`
- renamed `AirflowSensorDecorator.compile` to `AirflowSensorDecorator.validate`
- Checking if dagfile and flow file are same.
- fixing variable names.
- checking out `kubernetes_decorator.py` from master (6441ed5)
- bug fixing secret setting in airflow.
- simplified parameter type parsing logic
- refactoring airflow argument parsing code.
commit 83b20a7c6a13b3aedb7e603e139f07f0ef2fb646
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 13 14:02:57 2022 -0700
Addressing Final comments. (#57)
- Added dag-run timeout.
- airflow related scheduling checks in decorator.
- Auto naming sensors if no name is provided
- Annotations to k8s operators
- fix: argument serialization for `DAG` arguments (method names refactored like `to_dict` became `serialize`)
- annotation bug fix
- setting`workflow-timeout` for only scheduled dags
commit 4931f9c84e6a1d20fc3ecb41cf138b72e5dee629
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:50:49 2022 +0000
k8s bug fix
commit 200ae8ed4a00028f094281f73a939e7a4dcdf83a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:39:50 2022 +0000
removed un-used function
commit 70e285e9a7cfbec71fc293508a62c96f33562a01
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:38:37 2022 +0000
Removed unused `sanitize_label` function
commit 84fc622d8b11e718a849b2e2d91ceb3ea69917e6
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:37:34 2022 +0000
GPU support added + container naming same as argo
commit c92280d8796ec12b4ff17fa2ff3c736c7244f39c
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 6 04:25:17 2022 +0000
Refactored sensors to different files + bug fix
- bug caused due `util.compress_list`.
- The function doesn't play nice with strings with variety of characters.
- Ensured that exceptions are handled appropriately.
- Made new file for each sensor under `airflow.sensors` module.
commit b72a1dcf0dbbbcb814581d92738fd27ec31ef673
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Jun 4 01:41:49 2022 +0000
ran black.
commit 558c82f65b383ed0d61ded6bc80326471e284550
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 18:32:48 2022 -0700
Moving information from airflow_utils to compiler (#56)
- commenting todos to organize unfinished changes.
- some environment variables set via`V1EnvVar`
- `client.V1ObjectFieldSelector` mapped env vars were not working in json form
- Moving k8s operator import into its own function.
- env vars moved.
commit 9bb5f638792a671164ec95891e97f599e9a3385f
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 18:06:03 2022 +0000
added mising Run-id prefixes to variables.
- merged `hash` and `dash_connect` filters.
commit 37b5e6a9d8ca93cc91244c8d77c7d4f61280ba59
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 18:00:22 2022 +0000
nit fix : variable name change.
commit 660756f952ebd92ba1e26d7f908b81036c31ff10
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 17:58:34 2022 +0000
nit fixes to dag.py's templating variables.
commit 1202f5bc92f76df52b5957f11c8574cadfa62196
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 17:56:53 2022 +0000
Fixed defaults passing
- Addressed comments for airflow.py
commit b9387dd428c1a37f9a3bfe2c72cab475da708c02
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 17:52:24 2022 +0000
Following Changes:
- Refactors setting scheduling interval
- refactor dag file creating function
- refactored is_active to is_paused_upon_creation
- removed catchup
commit 054e3f389febc6c447494a1dedb01228f5f5650f
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 17:33:25 2022 +0000
Multiple Changes based on comments:
1. refactored `create_k8s_args` into _to_job
2. Addressed comments for snake casing
3. refactored `attrs` for simplicity.
4. refactored `metaflow_parameters` to `parameters`.
5. Refactored setting of `input_paths`
commit d481b2fca7914b6b657a69af407cfe1a894a46dc
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 16:42:24 2022 +0000
Removed Sensor metadata extraction.
commit d8e6ec044ef8c285d7fbe1b83c10c07d51c063e3
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Jun 3 16:30:34 2022 +0000
porting savin's comments
- next changes : addressing comments.
commit 3f2353a647e53bc240e28792769c42a71ea8f8c9
Merge: d370ffb c1ff469
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Jul 28 23:52:16 2022 +0000
Merge branch 'master' into airflow
commit d370ffb248411ad4675f9d55de709dbd75d3806e
Merge: a82f144 e4eb751
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Jul 14 19:38:48 2022 +0000
Merge branch 'master' into airflow
commit a82f1447b414171fc5611758cb6c12fc692f55f9
Merge: bdb1f0d 6f097e3
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Jul 13 00:35:49 2022 +0000
Merge branch 'master' into airflow
commit bdb1f0dd248d01318d4a493c75b6f54248c7be64
Merge: 8511215 f9a4968
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Jun 29 18:44:51 2022 +0000
Merge branch 'master' into airflow
commit 85112158cd352cb7de95a2262c011c6f43d98283
Author: Valay Dave <valaygaurang@gmail.com>
Date: Tue Jun 21 02:53:11 2022 +0000
Bug fix from master merge.
commit 90c06f12bb14eda51c6a641766c5f67d6763abaa
Merge: 0fb73af 6441ed5
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon Jun 20 21:20:20 2022 +0000
Merge branch 'master' into airflow
commit 0fb73af8af9fca2875261e3bdd305a0daab1b229
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Jun 4 00:53:10 2022 +0000
squashing bugs after changes from master.
commit 09c6ba779f6b1b6ef1d7ed5b1bb2be70ec76575d
Merge: 7bdf662 ffff49b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Sat Jun 4 00:20:38 2022 +0000
Merge branch 'master' into af-mmr
commit 7bdf662e14966b929b8369c65d5bd3bbe5741937
Author: Valay Dave <valaygaurang@gmail.com>
Date: Mon May 16 17:42:38 2022 -0700
Airflow sensor api (#3)
* Fixed run-id setting
- Change gaurentees that multiple dags triggered at same moment have unique run-id
* added allow multiple in `Decorator` class
* Airflow sensor integration.
>> support added for :
- ExternalTaskSensor
- S3KeySensor
- SqlSensor
>> sensors allow multiple decorators
>> sensors accept those arguments which are supported by airflow
* Added `@airflow_schedule_interval` decorator
* Fixing bug run-id related in env variable setting.
commit 2604a29452e794354cf4c612f48bae7cf45856ee
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 21 18:26:59 2022 +0000
Addressed comments.
commit 584e88b679fed7d6eec8ce564bf3707359170568
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:33:55 2022 +0000
fixed printing bug
commit 169ac1535e5567149d94749ddaf70264e882d62c
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:30:59 2022 +0000
Option help bug fix.
commit 6f8489bcc3bd715b65d8a8554a0f3932dc78c6f5
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:25:54 2022 +0000
variable renamemetaflow_specific_args
commit 0c779abcd1d9574878da6de8183461b53e0da366
Merge: d299b13 5a61508
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:23:10 2022 +0000
Merge branch 'airflow-tests' into airflow
commit 5a61508e61583b567ef8d3fea04e049d74a6d973
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:22:54 2022 +0000
Removing un-used code / resolved-todos.
commit d030830f2543f489a1c4ebd17da1b47942f041d6
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:06:03 2022 +0000
ran black,
commit 2d1fc06e41cbe45ccfd46e03bc87b09c7a78da45
Merge: f2cb319 7921d13
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:04:19 2022 +0000
Merge branch 'master' into airflow-tests
commit d299b13ce38d027ab27ce23c9bbcc0f43b222cfa
Merge: f2cb319 7921d13
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 03:02:37 2022 +0000
Merge branch 'master' into airflow
commit f2cb3197725f11520da0d49cbeef8de215c243eb
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 02:54:03 2022 +0000
reverting change.
commit 05b9db9cf0fe8b40873b2b74e203b4fc82e7fea4
Author: Valay Dave <valaygaurang@gmail.com>
Date: Wed Apr 20 02:47:41 2022 +0000
3 changes:
- Removing s3 dep
- remove uesless import
- added `deployed_on` in dag file template
commit c6afba95f5ec05acf7f33fd3228cffd784556e3b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Apr 15 22:50:52 2022 +0000
Fixed passing secrets with kubernetes.
commit c3ce7e9faa5f7a23d309e2f66f778dbca85df22a
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Apr 15 22:04:22 2022 +0000
Refactored code .
- removed compute/k8s.py
- Moved k8s code to airflow_compiler.py
- ran isort to airflow_compiler.py
commit d1c343dbbffbddbebd2aeda26d6846e595144e0b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Apr 15 18:02:25 2022 +0000
Added validations about:
- un-supported decorators
- foreach
Changed where validations are done to not save the package.
commit 7b19f8e66e278c75d836daf6a1c7ed2c607417ce
Author: Valay Dave <valaygaurang@gmail.com>
Date: Fri Apr 15 03:34:26 2022 +0000
Fixing mf log related bug
- No double logging on metaflow.
commit 4d1f6bf9bb32868c949d8c103c8fe44ea41b3f13
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 03:10:51 2022 +0000
Removed usless code WRT project decorator.
commit 5ad9a3949e351b0ac13f11df13446953932e8ffc
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 03:03:19 2022 +0000
Remove readme.
commit 60cb6a79404efe2bcf9bf9a118a68f0b98c7d771
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 03:02:38 2022 +0000
Made file path required arguement.
commit 9f0dc1b2e01ee04b05620630f3a0ec04fe873a31
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 03:01:07 2022 +0000
changed `--is-active`->`--is-paused-upon-creation`
- dags are active by default.
commit 5b98f937a62ee74de8aed8b0efde5045a28f068b
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 02:55:46 2022 +0000
shortened length of run-id and task-id hashes.
commit e53426eaa4b156e8bd70ae7510c2e7c66745d101
Author: Valay Dave <valaygaurang@gmail.com>
Date: Thu Apr 14 02:41:32 2022 +0000
Remov…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces Metaflow's integration with Argo Workflows (#50)
The integration works very similarly to the current integration with AWS Step-Functions. You can take any arbitrary Metaflow flow and dispatch it to Argo Workflows through a single command
You can trigger the execution of the said workflow either through the UI or through the command
The documentation in Scheduling Metaflow Flows and Coordinating Larger Metaflow Projects applies to this integration in its entirety. Just replace the AWS Step Functions references with Argo Workflows.
All Metaflow features now work within Argo Workflows. If you are using/considering Kubernetes for your ML needs, do give this a try!
This functionality requires -
METAFLOW_KUBERNETES_SERVICE_ACCOUNTon your workstation.This builds upon the fantastic work done by @sappier and his team at SAP in #434.
Ping us at chat.metaflow.org for any questions/feedback.
User documentation