Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for hosting artifacts as microservices #3

Open
romain-intel opened this issue Dec 2, 2019 · 11 comments
Open

Support for hosting artifacts as microservices #3

romain-intel opened this issue Dec 2, 2019 · 11 comments
Assignees
Labels
enhancement New feature or request

Comments

@romain-intel
Copy link
Contributor

romain-intel commented Dec 2, 2019

Once Metaflow has been used to train a model, it produces artifacts that are typically persisted (for example in S3). A natural extension of this is to provide an easy mechanism to deploy web services that would take these artifacts and serve them in some way so that they can be consumed by downstream applications.

@romain-intel romain-intel added the enhancement New feature or request label Dec 2, 2019
@wuqunfei
Copy link

wuqunfei commented Dec 4, 2019

`from metaflow import WebServiceSpec
from metaflow import endpoint

class MyWebService(WebServiceSpec):

@ednpoint
def show_data(self, request):
### TODO
return {}`

I did not find these classes in master branch. looking forward to seeing.

@savingoyal
Copy link
Collaborator

savingoyal commented Dec 4, 2019

@wuqunfei Yes, you are correct, we haven’t released WebServiceSpec yet.

@wuqunfei
Copy link

wuqunfei commented Dec 4, 2019

thanks, @savingoyal, it is a really good idea to expose endpoint. I am using sagemaker to host endpoint, but it is not so easy as your annotation.

@justinttl
Copy link

justinttl commented Dec 12, 2019

Hi! Saw the talk on this topic yesterday and am really excited for this feature to be released in the future.

Is there a channel where development on this is being discussed? Would love to participate and help out.

@savingoyal
Copy link
Collaborator

savingoyal commented Dec 12, 2019

Yes, @justinttl this issue tracks the development.

@romain-intel
Copy link
Contributor Author

romain-intel commented Dec 13, 2019

This is a good medium to communicate your ideas, etc. We are currently working on a redesign of the ideas expressed in the talk internally and, given sufficient interest, we will look at releasing this to the community as well. The talk (and associated slides) give a pretty good idea of where we are at right now. Comments/thoughts are more than welcome here!

@retnuh
Copy link

retnuh commented Dec 30, 2019

Kubeflow uses seldon.io for deploying models; it may be worth looking into - or perhaps a pluggable layer with a seldon backend.

@sl-victormazzeo
Copy link

sl-victormazzeo commented Mar 5, 2020

Hi @romain-intel , any information on the progress of this feature? Thanks!

@agusgun
Copy link

agusgun commented Mar 24, 2020

How about we package the model artifact and deploy it using Sagemaker in the end step?

@brightsparc
Copy link

brightsparc commented May 6, 2020

It might be worth exploring using the Amazon SageMaker Training Toolkit as a way to create a managed docker container in Amazon SageMaker for a particular step.

@savingoyal
Copy link
Collaborator

savingoyal commented May 19, 2020

@sl-victormazzeo We are looking into publishing recipes on how to deploy metaflow artifacts within a micro-service, independent of this effort. Today, you can access your artifacts via the metaflow client within hosting services like Seldon, AWS Lambda etc.

@agusgun Yes, you can very simply use boto3 to package your model artifact and deploy it to Sagemaker in the end step.

@brightsparc We do have a pending PR that tries to achieve something very similar.

savingoyal pushed a commit that referenced this issue Aug 3, 2022
* Airflow on Kubernetes minus Foreachs.
- Support for all metaflow construct without foreach and sensors

Squashed commit of the following:

commit ef8b1e3
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jul 29 01:06:26 2022 +0000

    Removed sernsors and banned foreach's

commit 8d517c4
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jul 29 00:59:01 2022 +0000

    commiting k8s related file from master.

commit a7e1ecd
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jul 29 00:54:45 2022 +0000

    Uncommented code for foreach support with k8s

    KubernetesPodOperator version 4.2.0 renamed `resources` to
    `container_resources`
    - Check : (apache/airflow#24673) /
    - (apache/airflow@45f4290)

    This was done because `KubernetesPodOperator` didn't play nice with dynamic task mapping and they had to deprecate the `resources` argument. Hence the below codepath checks for the version of `KubernetesPodOperator`
    and then sets the argument. If the version < 4.2.0 then we set the argument as `resources`.
    If it is > 4.2.0 then we set the argument as `container_resources`
    The `resources` argument of KuberentesPodOperator is going to be deprecated soon in the future.
    So we will only use it for `KuberentesPodOperator` version < 4.2.0
    The `resources` argument will also not work for foreach's.

commit 2719f5d
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jul 18 18:31:58 2022 +0000

    nit fixes :
    - fixing comments.
    - refactor some variable/function names.

commit 2079293
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jul 18 18:14:53 2022 +0000

    change `token` to `production_token`

commit 14aad5f
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jul 18 18:11:56 2022 +0000

    Refactored import Airflow Sensors.

commit b1472d5
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jul 18 18:08:41 2022 +0000

    new comment on `startup_timeout_seconds` env var.

commit 6d81b75
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jul 18 18:06:09 2022 +0000

    Removing traces of `@airflow_schedule_interval`

commit 0673db7
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Jul 14 12:43:08 2022 -0700

    Foreach polish (valayDave#62)

    * Removing unused imports
    * Added validation logic for airflow version numbers with foreaches
    * Removed `airflow_schedule_interval` decorator.

    * Added production/deployment token related changes
    - Uses s3 as a backend to store the production token
    - Token used for avoiding nameclashes
    - token stored via `FlowDatastore`

    * Graph type validation for airflow foreachs
    - Airflow foreachs only support single node fanout.
    - validation invalidates graphs with nested foreachs

    * Added configuration about startup_timeout.

    * Added final todo on `resources` argument of k8sOp
    - added a commented code block
    - it needs to be uncommented when airflow releasese the patch for the op
    - Code seems feature complete keeping aside airflow patch

commit 4b2dd12
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Jul 7 19:33:07 2022 +0000

    Removed retries from user-defaults.

commit 0e87a97
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Jul 6 16:29:33 2022 +0000

    updated pod startup time

commit fce2bd2
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Jun 29 18:44:11 2022 +0000

    Adding default 1 retry for any airflow worker.

commit 5ef6bbc
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 27 01:22:42 2022 +0000

    Airflow Foreach Integration
    - Simple one node foreach-join support as gaurenteed by airflow
    - Fixed env variable setting issue
    - introduced MetaflowKuberentesOperator
    - Created a new operator to allow smootness in plumbing xcom values
    - Some todos

commit d319fa9
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 24 21:12:09 2022 +0000

    simplifying run-id macro.

commit 0ffc813
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 24 11:51:42 2022 -0700

    Refactored parameter macro settings. (valayDave#60)

commit a3a4950
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 24 02:05:57 2022 +0000

    added comment on need for `start_date`

commit a3147be
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Tue Jun 21 06:03:56 2022 +0000

    Refactored an `id_creator` method.

commit 04d7f20
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Tue Jun 21 05:52:05 2022 +0000

    refactor :
    -`RUN_ID_LEN` to `RUN_HASH_ID_LEN`
    - `TASK_ID_LEN` to `TASK_ID_HASH_LEN`

commit cde4605
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Tue Jun 21 05:48:55 2022 +0000

    refactored an error string

commit 1145818
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 20 22:42:36 2022 -0700

    addressing  savins comments. (#59)

    - Added many adhoc changes based for some comments.
    - Integrated secrets and `KUBERNETES_SECRETS`
    - cleaned up parameter setting
    - cleaned up setting of scheduling interval
    - renamed `AIRFLOW_TASK_ID_TEMPLATE_VALUE` to `AIRFLOW_TASK_ID`
    - renamed `AirflowSensorDecorator.compile` to `AirflowSensorDecorator.validate`
    - Checking if dagfile and flow file are same.
    - fixing variable names.
    - checking out `kubernetes_decorator.py` from master (6441ed5)
    - bug fixing secret setting in airflow.
    - simplified parameter type parsing logic
    - refactoring airflow argument parsing code.

commit 83b20a7
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 13 14:02:57 2022 -0700

    Addressing Final comments.  (#57)

    - Added dag-run timeout.
    - airflow related scheduling checks in decorator.
    - Auto naming sensors if no name is provided
    - Annotations to k8s operators
    - fix: argument serialization for `DAG` arguments (method names refactored like `to_dict` became `serialize`)
    - annotation bug fix
    - setting`workflow-timeout` for only scheduled dags

commit 4931f9c
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 6 04:50:49 2022 +0000

    k8s bug fix

commit 200ae8e
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 6 04:39:50 2022 +0000

    removed un-used function

commit 70e285e
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 6 04:38:37 2022 +0000

    Removed unused `sanitize_label` function

commit 84fc622
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 6 04:37:34 2022 +0000

    GPU support added + container naming same as argo

commit c92280d
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 6 04:25:17 2022 +0000

    Refactored sensors to different files + bug fix
    - bug caused due `util.compress_list`.
    - The function doesn't play nice with strings with variety of characters.
    - Ensured that exceptions are handled appropriately.
    - Made new file for each sensor under `airflow.sensors` module.

commit b72a1dc
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Jun 4 01:41:49 2022 +0000

    ran black.

commit 558c82f
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 18:32:48 2022 -0700

    Moving information from airflow_utils to compiler (#56)

    - commenting todos to organize unfinished changes.
    - some environment variables set via`V1EnvVar`
        - `client.V1ObjectFieldSelector` mapped env vars were not working in json form
        - Moving k8s operator import into its own function.
        - env vars moved.

commit 9bb5f63
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 18:06:03 2022 +0000

    added mising Run-id prefixes to variables.
    - merged `hash` and `dash_connect` filters.

commit 37b5e6a
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 18:00:22 2022 +0000

    nit fix : variable name change.

commit 660756f
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 17:58:34 2022 +0000

    nit fixes to dag.py's templating variables.

commit 1202f5b
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 17:56:53 2022 +0000

    Fixed defaults passing
    - Addressed comments for airflow.py

commit b9387dd
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 17:52:24 2022 +0000

    Following Changes:
    - Refactors setting scheduling interval
    - refactor dag file creating function
    - refactored is_active to is_paused_upon_creation
    - removed catchup

commit 054e3f3
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 17:33:25 2022 +0000

    Multiple Changes based on comments:
    1. refactored `create_k8s_args` into _to_job
    2. Addressed comments for snake casing
    3. refactored `attrs` for simplicity.
    4. refactored `metaflow_parameters` to `parameters`.
    5. Refactored setting of `input_paths`

commit d481b2f
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 16:42:24 2022 +0000

    Removed Sensor metadata extraction.

commit d8e6ec0
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 16:30:34 2022 +0000

    porting savin's comments
    - next changes : addressing comments.

commit 3f2353a
Merge: d370ffb c1ff469
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Jul 28 23:52:16 2022 +0000

    Merge branch 'master' into airflow

commit d370ffb
Merge: a82f144 e4eb751
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Jul 14 19:38:48 2022 +0000

    Merge branch 'master' into airflow

commit a82f144
Merge: bdb1f0d 6f097e3
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Jul 13 00:35:49 2022 +0000

    Merge branch 'master' into airflow

commit bdb1f0d
Merge: 8511215 f9a4968
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Jun 29 18:44:51 2022 +0000

    Merge branch 'master' into airflow

commit 8511215
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Tue Jun 21 02:53:11 2022 +0000

    Bug fix from master merge.

commit 90c06f1
Merge: 0fb73af 6441ed5
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 20 21:20:20 2022 +0000

    Merge branch 'master' into airflow

commit 0fb73af
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Jun 4 00:53:10 2022 +0000

    squashing bugs after changes from master.

commit 09c6ba7
Merge: 7bdf662 ffff49b
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Jun 4 00:20:38 2022 +0000

    Merge branch 'master' into af-mmr

commit 7bdf662
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon May 16 17:42:38 2022 -0700

    Airflow sensor api (#3)

    * Fixed run-id setting
    - Change gaurentees that multiple dags triggered at same moment have unique run-id

    * added allow multiple in `Decorator` class

    * Airflow sensor integration.
     >> support added for :
    - ExternalTaskSensor
    - S3KeySensor
    - SqlSensor
    >> sensors allow multiple decorators
    >> sensors accept those arguments which are supported by airflow

    * Added `@airflow_schedule_interval` decorator
    * Fixing bug run-id related in env variable setting.

commit 2604a29
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 21 18:26:59 2022 +0000

    Addressed comments.

commit 584e88b
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 03:33:55 2022 +0000

    fixed printing bug

commit 169ac15
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 03:30:59 2022 +0000

    Option help bug fix.

commit 6f8489b
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 03:25:54 2022 +0000

    variable renamemetaflow_specific_args

commit 0c779ab
Merge: d299b13 5a61508
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 03:23:10 2022 +0000

    Merge branch 'airflow-tests' into airflow

commit 5a61508
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 03:22:54 2022 +0000

    Removing un-used code / resolved-todos.

commit d030830
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 03:06:03 2022 +0000

    ran black,

commit 2d1fc06
Merge: f2cb319 7921d13
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 03:04:19 2022 +0000

    Merge branch 'master' into airflow-tests

commit d299b13
Merge: f2cb319 7921d13
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 03:02:37 2022 +0000

    Merge branch 'master' into airflow

commit f2cb319
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 02:54:03 2022 +0000

    reverting change.

commit 05b9db9
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 02:47:41 2022 +0000

    3 changes:
    - Removing s3 dep
    - remove uesless import
    - added `deployed_on` in dag file template

commit c6afba9
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Apr 15 22:50:52 2022 +0000

    Fixed passing secrets with kubernetes.

commit c3ce7e9
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Apr 15 22:04:22 2022 +0000

    Refactored code .
    - removed compute/k8s.py
    - Moved k8s code to airflow_compiler.py
    - ran isort to airflow_compiler.py

commit d1c343d
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Apr 15 18:02:25 2022 +0000

    Added validations about:
    - un-supported decorators
    - foreach
    Changed where validations are done to not save the package.

commit 7b19f8e
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Apr 15 03:34:26 2022 +0000

    Fixing mf log related bug
    - No double logging on metaflow.

commit 4d1f6bf
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 14 03:10:51 2022 +0000

    Removed usless code WRT project decorator.

commit 5ad9a39
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 14 03:03:19 2022 +0000

    Remove readme.

commit 60cb6a7
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 14 03:02:38 2022 +0000

    Made file path required arguement.

commit 9f0dc1b
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 14 03:01:07 2022 +0000

    changed `--is-active`->`--is-paused-upon-creation`
    - dags are active by default.

commit 5b98f93
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 14 02:55:46 2022 +0000

    shortened length of run-id and task-id hashes.

commit e53426e
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 14 02:41:32 2022 +0000

    Removing un-used args.

commit 72cbbfc
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 14 02:39:59 2022 +0000

    Moved exceptions to airflow compiler

commit b2970dd
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 14 02:33:02 2022 +0000

    Changes based on PR comments:
    - removed airflow xcom push file , moved to decorator code
    - removed prefix configuration
    - nit fixes.

commit 9e622ba
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Apr 11 20:39:00 2022 +0000

    Removing un-used code paths + code cleanup

commit 7425f62
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Apr 11 19:45:04 2022 +0000

    Fixing bug fix in schedule.

commit eb775cb
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Apr 10 02:52:59 2022 +0000

    Bug fixes WRT Kubernetes secrets + k8s deployments.
    - Fixing some error messages.
    - Added some comments.

commit 04c92b9
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Apr 10 01:20:53 2022 +0000

    Added secrets support.

commit 4a0a85d
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Apr 10 00:11:46 2022 +0000

    Bug fix.

commit af91099
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Apr 10 00:03:34 2022 +0000

    bug fix.

commit c17f04a
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Apr 9 23:55:41 2022 +0000

    Bug fix in active defaults.

commit 0d37236
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Apr 9 23:54:02 2022 +0000

    @project, @schedule, default active dag support.
    - Added a flag to allow setting dag as active on creation
    - Airflow compatible schedule interval
    - Project name fixes.

commit 5c97b15
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 7 21:15:18 2022 +0000

    Max workers and worker pool support.

commit 9c973f2
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 7 19:34:33 2022 +0000

    Adding exceptions for missing features.

commit 2a946e2
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Mar 28 19:34:11 2022 +0000

    2 changes :
    - removed hacky line
    - added support to directly throw dags in s3.

commit e0772ec
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Mar 23 22:38:20 2022 +0000

    fixing bugs in service account setting

commit 874b94a
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Mar 20 23:49:15 2022 +0000

    Added support for Branching with Airflow
    - remove `next` function in `AirflowTask`
    - `AirflowTask`s has no knowledge of next tasks.
    - removed todos + added some todos
    - Graph construction on airflow side using graph_structure datastructure.
    - graph_structure comes from`FlowGraph.output_steps()[1]`

commit 8e9f649
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Mar 20 02:33:04 2022 +0000

    Added hacky line

commit fd5db04
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Mar 20 02:06:38 2022 +0000

    Removed hacky line.

commit 5b23eb7
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Mar 20 01:44:57 2022 +0000

    Added support for Parameters.
    - Supporting int, str, bool, float, JSONType

commit c9378e9
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Mar 20 00:14:10 2022 +0000

    Removed todos + added some validation logic.

commit 7250a44
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Mar 19 23:45:15 2022 +0000

    Fixing logs related change from master.

commit d125978
Merge: 8cdac53 7e210a2
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Mar 19 23:42:48 2022 +0000

    Merge branch 'master' into aft-mm

commit 8cdac53
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Mar 19 23:36:47 2022 +0000

    making changes sync with master

commit 5a93d9f
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Mar 19 23:29:47 2022 +0000

    Fixed bug when using catch + retry

commit 62bc8df
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Mar 19 22:58:37 2022 +0000

    Changed retry setting.

commit 563a200
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Mar 19 22:42:57 2022 +0000

    Fixed setting `task_id` :
    - switch task-id from airflow job is to hash to "runid/stepname"
    - refactor xcom setting variables
    - added comments

commit e2a1e50
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Mar 19 17:51:59 2022 +0000

    setting retry logic.

commit a697b56
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Mar 17 01:02:11 2022 +0000

    Nit fix.

commit 68f13be
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Mar 16 20:46:19 2022 +0000

    Added @schedule support + readme

commit 57bdde5
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Tue Mar 15 19:47:06 2022 +0000

    Fixed setting run-id / task-id to labels in k8s
    - Fixed setting run-id has from cli macro
    - added hashing macro to ensure that jinja template set the correct run-id to k8s labels
    -

commit 3d6c319
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Tue Mar 15 05:39:04 2022 +0000

    Got linear workflows working on airflow.
    - Still not feature complete as lots of args are still unfilled / lots of unknows
    - minor tweek in eks to ensure airflow is k8s compatible.
    - passing state around via xcom-push
    - HACK : AWS keys are passed in a shady way. : Reverse this soon.

commit db074b8
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Mar 11 12:34:33 2022 -0800

    Tweeks

commit a9f0468
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Tue Mar 1 17:14:47 2022 -0800

    some changes based on savin's comments.
    - Added changes to task datastore for different reason : (todo) Decouple these
    - Added comments to SFN for reference.
    - Airflow DAG is no longer dependent on metaflow

commit f32d089
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Feb 23 00:54:17 2022 -0800

    First version of dynamic dag compiler.
    - Not completely finished code
    - Creates generic .py file a JSON that is parsed to create Airflow DAG.
    - Currently only boiler plate to make a linear dag but doesn't execute anything.
    -  Unfinished code.

commit d2def66
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Feb 19 14:01:47 2022 -0800

    more tweeks.

commit b176311
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Feb 17 09:04:29 2022 -0800

    commit 0
    - unfinished code.

* Making version compatibility changes.
- Minimum support version to 2.2.0

* Task-id macro related logic refactor
- Done for better version support.

* bug fix: param related task-id setting.

* applied black

* Reverting `decorators.py` to master
savingoyal added a commit that referenced this issue Aug 23, 2022
* Azure Storage

* fix BrokenProcessPool import for older pythons

* Fix batch

* add azure configure, cannot validate storage account url after all (catch-22 in configure)

* rename storage account url to AZURE_STORAGE_BLOB_SERVICE_ENDPOINT

* fix indent

* clean up kubernetes config cli, add secrets

* remove signature

* fix fractional resoure handling for batch (#1089)

* bump version (#1093)

* Fix docstrings for the API reference (no functional changes!) (#1076)

Make the docstring format in public APIs compliant with the new API reference framework

* Fix issue with get_pinned_conda_libs and metaflow extensions

* clean up includefile a bit, remove from_env (not used)

* Move a sys.path modification in s3op to __main__ (#1095)

In its current location, this could modify the sys.path of the current
running metaflow which could have nefarious consequences with the escape
hatch which uses sys.path to determine the outside environment's python path.

The following scenario would cause issues:
  - metaflow is installed in the usual path on the system
  - a conda environment was manually bootstrapped from a directory A.
    + at this point, sys.path starts with `A` and then contains the other
      system includes
    + s3op.py is imported at some point by the Conda installer when it calls `get_many`
    + this modifies sys.path to insert, at the beginning, the parent of Metaflow; so
      in this case, sys.path looks something like
      ['/apps/python3/lib/python3.7/site-packages', 'A', '/apps/python3/lib/python3.7/site-packages'...]
    + when the escape hatch trampolines are created, this sys.path is used to determine what the
      sys.path for the outside interpreter is.
    + in A, we create:
      * INFO
      * metaflow
      * metaflow_extensions
      which properly describe the installation of metaflow
  - when the escape hatch client runs, it runs in the conda environment and uses metaflow
    created in A.
  - when the client wants to start the server, this is where we run into issues because, at this
    point, the server will use the PYTHONPATH which starts with '/apps/python3/lib/python3.7/site-packages'
    in which it will find metaflow. It will therefore use that metaflow (which is the same
    as the one linked in A) to start the server. This runs into issues though because A
    is also in PYTHONPATH and so the extension support loader will also try to load `A/metaflow_extensions`.
    This will cause issues if multiple extensions are installed there (it will complain about duplicate
    configurations for example. The `INFO` file typically used to solve this problem is not read as it was
    not present for the TL metaflow.

This patch simply moves the modification of sys.path to where it is actually needed and avoids polluting
sys.path when the module is simply included (and not called as a script).

* Airflow Support (#1094)

* Airflow on Kubernetes minus Foreachs.
- Support for all metaflow construct without foreach and sensors

Squashed commit of the following:

commit ef8b1e3
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jul 29 01:06:26 2022 +0000

    Removed sernsors and banned foreach's

commit 8d517c4
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jul 29 00:59:01 2022 +0000

    commiting k8s related file from master.

commit a7e1ecd
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jul 29 00:54:45 2022 +0000

    Uncommented code for foreach support with k8s

    KubernetesPodOperator version 4.2.0 renamed `resources` to
    `container_resources`
    - Check : (apache/airflow#24673) /
    - (apache/airflow@45f4290)

    This was done because `KubernetesPodOperator` didn't play nice with dynamic task mapping and they had to deprecate the `resources` argument. Hence the below codepath checks for the version of `KubernetesPodOperator`
    and then sets the argument. If the version < 4.2.0 then we set the argument as `resources`.
    If it is > 4.2.0 then we set the argument as `container_resources`
    The `resources` argument of KuberentesPodOperator is going to be deprecated soon in the future.
    So we will only use it for `KuberentesPodOperator` version < 4.2.0
    The `resources` argument will also not work for foreach's.

commit 2719f5d
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jul 18 18:31:58 2022 +0000

    nit fixes :
    - fixing comments.
    - refactor some variable/function names.

commit 2079293
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jul 18 18:14:53 2022 +0000

    change `token` to `production_token`

commit 14aad5f
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jul 18 18:11:56 2022 +0000

    Refactored import Airflow Sensors.

commit b1472d5
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jul 18 18:08:41 2022 +0000

    new comment on `startup_timeout_seconds` env var.

commit 6d81b75
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jul 18 18:06:09 2022 +0000

    Removing traces of `@airflow_schedule_interval`

commit 0673db7
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Jul 14 12:43:08 2022 -0700

    Foreach polish (valayDave#62)

    * Removing unused imports
    * Added validation logic for airflow version numbers with foreaches
    * Removed `airflow_schedule_interval` decorator.

    * Added production/deployment token related changes
    - Uses s3 as a backend to store the production token
    - Token used for avoiding nameclashes
    - token stored via `FlowDatastore`

    * Graph type validation for airflow foreachs
    - Airflow foreachs only support single node fanout.
    - validation invalidates graphs with nested foreachs

    * Added configuration about startup_timeout.

    * Added final todo on `resources` argument of k8sOp
    - added a commented code block
    - it needs to be uncommented when airflow releasese the patch for the op
    - Code seems feature complete keeping aside airflow patch

commit 4b2dd12
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Jul 7 19:33:07 2022 +0000

    Removed retries from user-defaults.

commit 0e87a97
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Jul 6 16:29:33 2022 +0000

    updated pod startup time

commit fce2bd2
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Jun 29 18:44:11 2022 +0000

    Adding default 1 retry for any airflow worker.

commit 5ef6bbc
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 27 01:22:42 2022 +0000

    Airflow Foreach Integration
    - Simple one node foreach-join support as gaurenteed by airflow
    - Fixed env variable setting issue
    - introduced MetaflowKuberentesOperator
    - Created a new operator to allow smootness in plumbing xcom values
    - Some todos

commit d319fa9
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 24 21:12:09 2022 +0000

    simplifying run-id macro.

commit 0ffc813
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 24 11:51:42 2022 -0700

    Refactored parameter macro settings. (valayDave#60)

commit a3a4950
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 24 02:05:57 2022 +0000

    added comment on need for `start_date`

commit a3147be
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Tue Jun 21 06:03:56 2022 +0000

    Refactored an `id_creator` method.

commit 04d7f20
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Tue Jun 21 05:52:05 2022 +0000

    refactor :
    -`RUN_ID_LEN` to `RUN_HASH_ID_LEN`
    - `TASK_ID_LEN` to `TASK_ID_HASH_LEN`

commit cde4605
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Tue Jun 21 05:48:55 2022 +0000

    refactored an error string

commit 1145818
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 20 22:42:36 2022 -0700

    addressing  savins comments. (#59)

    - Added many adhoc changes based for some comments.
    - Integrated secrets and `KUBERNETES_SECRETS`
    - cleaned up parameter setting
    - cleaned up setting of scheduling interval
    - renamed `AIRFLOW_TASK_ID_TEMPLATE_VALUE` to `AIRFLOW_TASK_ID`
    - renamed `AirflowSensorDecorator.compile` to `AirflowSensorDecorator.validate`
    - Checking if dagfile and flow file are same.
    - fixing variable names.
    - checking out `kubernetes_decorator.py` from master (6441ed5)
    - bug fixing secret setting in airflow.
    - simplified parameter type parsing logic
    - refactoring airflow argument parsing code.

commit 83b20a7
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 13 14:02:57 2022 -0700

    Addressing Final comments.  (#57)

    - Added dag-run timeout.
    - airflow related scheduling checks in decorator.
    - Auto naming sensors if no name is provided
    - Annotations to k8s operators
    - fix: argument serialization for `DAG` arguments (method names refactored like `to_dict` became `serialize`)
    - annotation bug fix
    - setting`workflow-timeout` for only scheduled dags

commit 4931f9c
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 6 04:50:49 2022 +0000

    k8s bug fix

commit 200ae8e
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 6 04:39:50 2022 +0000

    removed un-used function

commit 70e285e
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 6 04:38:37 2022 +0000

    Removed unused `sanitize_label` function

commit 84fc622
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 6 04:37:34 2022 +0000

    GPU support added + container naming same as argo

commit c92280d
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 6 04:25:17 2022 +0000

    Refactored sensors to different files + bug fix
    - bug caused due `util.compress_list`.
    - The function doesn't play nice with strings with variety of characters.
    - Ensured that exceptions are handled appropriately.
    - Made new file for each sensor under `airflow.sensors` module.

commit b72a1dc
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Jun 4 01:41:49 2022 +0000

    ran black.

commit 558c82f
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 18:32:48 2022 -0700

    Moving information from airflow_utils to compiler (#56)

    - commenting todos to organize unfinished changes.
    - some environment variables set via`V1EnvVar`
        - `client.V1ObjectFieldSelector` mapped env vars were not working in json form
        - Moving k8s operator import into its own function.
        - env vars moved.

commit 9bb5f63
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 18:06:03 2022 +0000

    added mising Run-id prefixes to variables.
    - merged `hash` and `dash_connect` filters.

commit 37b5e6a
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 18:00:22 2022 +0000

    nit fix : variable name change.

commit 660756f
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 17:58:34 2022 +0000

    nit fixes to dag.py's templating variables.

commit 1202f5b
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 17:56:53 2022 +0000

    Fixed defaults passing
    - Addressed comments for airflow.py

commit b9387dd
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 17:52:24 2022 +0000

    Following Changes:
    - Refactors setting scheduling interval
    - refactor dag file creating function
    - refactored is_active to is_paused_upon_creation
    - removed catchup

commit 054e3f3
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 17:33:25 2022 +0000

    Multiple Changes based on comments:
    1. refactored `create_k8s_args` into _to_job
    2. Addressed comments for snake casing
    3. refactored `attrs` for simplicity.
    4. refactored `metaflow_parameters` to `parameters`.
    5. Refactored setting of `input_paths`

commit d481b2f
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 16:42:24 2022 +0000

    Removed Sensor metadata extraction.

commit d8e6ec0
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Jun 3 16:30:34 2022 +0000

    porting savin's comments
    - next changes : addressing comments.

commit 3f2353a
Merge: d370ffb c1ff469
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Jul 28 23:52:16 2022 +0000

    Merge branch 'master' into airflow

commit d370ffb
Merge: a82f144 e4eb751
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Jul 14 19:38:48 2022 +0000

    Merge branch 'master' into airflow

commit a82f144
Merge: bdb1f0d 6f097e3
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Jul 13 00:35:49 2022 +0000

    Merge branch 'master' into airflow

commit bdb1f0d
Merge: 8511215 f9a4968
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Jun 29 18:44:51 2022 +0000

    Merge branch 'master' into airflow

commit 8511215
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Tue Jun 21 02:53:11 2022 +0000

    Bug fix from master merge.

commit 90c06f1
Merge: 0fb73af 6441ed5
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Jun 20 21:20:20 2022 +0000

    Merge branch 'master' into airflow

commit 0fb73af
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Jun 4 00:53:10 2022 +0000

    squashing bugs after changes from master.

commit 09c6ba7
Merge: 7bdf662 ffff49b
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Jun 4 00:20:38 2022 +0000

    Merge branch 'master' into af-mmr

commit 7bdf662
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon May 16 17:42:38 2022 -0700

    Airflow sensor api (#3)

    * Fixed run-id setting
    - Change gaurentees that multiple dags triggered at same moment have unique run-id

    * added allow multiple in `Decorator` class

    * Airflow sensor integration.
     >> support added for :
    - ExternalTaskSensor
    - S3KeySensor
    - SqlSensor
    >> sensors allow multiple decorators
    >> sensors accept those arguments which are supported by airflow

    * Added `@airflow_schedule_interval` decorator
    * Fixing bug run-id related in env variable setting.

commit 2604a29
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 21 18:26:59 2022 +0000

    Addressed comments.

commit 584e88b
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 03:33:55 2022 +0000

    fixed printing bug

commit 169ac15
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 03:30:59 2022 +0000

    Option help bug fix.

commit 6f8489b
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 03:25:54 2022 +0000

    variable renamemetaflow_specific_args

commit 0c779ab
Merge: d299b13 5a61508
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 03:23:10 2022 +0000

    Merge branch 'airflow-tests' into airflow

commit 5a61508
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 03:22:54 2022 +0000

    Removing un-used code / resolved-todos.

commit d030830
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 03:06:03 2022 +0000

    ran black,

commit 2d1fc06
Merge: f2cb319 7921d13
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 03:04:19 2022 +0000

    Merge branch 'master' into airflow-tests

commit d299b13
Merge: f2cb319 7921d13
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 03:02:37 2022 +0000

    Merge branch 'master' into airflow

commit f2cb319
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 02:54:03 2022 +0000

    reverting change.

commit 05b9db9
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Apr 20 02:47:41 2022 +0000

    3 changes:
    - Removing s3 dep
    - remove uesless import
    - added `deployed_on` in dag file template

commit c6afba9
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Apr 15 22:50:52 2022 +0000

    Fixed passing secrets with kubernetes.

commit c3ce7e9
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Apr 15 22:04:22 2022 +0000

    Refactored code .
    - removed compute/k8s.py
    - Moved k8s code to airflow_compiler.py
    - ran isort to airflow_compiler.py

commit d1c343d
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Apr 15 18:02:25 2022 +0000

    Added validations about:
    - un-supported decorators
    - foreach
    Changed where validations are done to not save the package.

commit 7b19f8e
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Apr 15 03:34:26 2022 +0000

    Fixing mf log related bug
    - No double logging on metaflow.

commit 4d1f6bf
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 14 03:10:51 2022 +0000

    Removed usless code WRT project decorator.

commit 5ad9a39
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 14 03:03:19 2022 +0000

    Remove readme.

commit 60cb6a7
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 14 03:02:38 2022 +0000

    Made file path required arguement.

commit 9f0dc1b
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 14 03:01:07 2022 +0000

    changed `--is-active`->`--is-paused-upon-creation`
    - dags are active by default.

commit 5b98f93
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 14 02:55:46 2022 +0000

    shortened length of run-id and task-id hashes.

commit e53426e
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 14 02:41:32 2022 +0000

    Removing un-used args.

commit 72cbbfc
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 14 02:39:59 2022 +0000

    Moved exceptions to airflow compiler

commit b2970dd
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 14 02:33:02 2022 +0000

    Changes based on PR comments:
    - removed airflow xcom push file , moved to decorator code
    - removed prefix configuration
    - nit fixes.

commit 9e622ba
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Apr 11 20:39:00 2022 +0000

    Removing un-used code paths + code cleanup

commit 7425f62
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Apr 11 19:45:04 2022 +0000

    Fixing bug fix in schedule.

commit eb775cb
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Apr 10 02:52:59 2022 +0000

    Bug fixes WRT Kubernetes secrets + k8s deployments.
    - Fixing some error messages.
    - Added some comments.

commit 04c92b9
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Apr 10 01:20:53 2022 +0000

    Added secrets support.

commit 4a0a85d
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Apr 10 00:11:46 2022 +0000

    Bug fix.

commit af91099
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Apr 10 00:03:34 2022 +0000

    bug fix.

commit c17f04a
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Apr 9 23:55:41 2022 +0000

    Bug fix in active defaults.

commit 0d37236
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Apr 9 23:54:02 2022 +0000

    @project, @schedule, default active dag support.
    - Added a flag to allow setting dag as active on creation
    - Airflow compatible schedule interval
    - Project name fixes.

commit 5c97b15
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 7 21:15:18 2022 +0000

    Max workers and worker pool support.

commit 9c973f2
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Apr 7 19:34:33 2022 +0000

    Adding exceptions for missing features.

commit 2a946e2
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Mon Mar 28 19:34:11 2022 +0000

    2 changes :
    - removed hacky line
    - added support to directly throw dags in s3.

commit e0772ec
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Mar 23 22:38:20 2022 +0000

    fixing bugs in service account setting

commit 874b94a
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Mar 20 23:49:15 2022 +0000

    Added support for Branching with Airflow
    - remove `next` function in `AirflowTask`
    - `AirflowTask`s has no knowledge of next tasks.
    - removed todos + added some todos
    - Graph construction on airflow side using graph_structure datastructure.
    - graph_structure comes from`FlowGraph.output_steps()[1]`

commit 8e9f649
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Mar 20 02:33:04 2022 +0000

    Added hacky line

commit fd5db04
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Mar 20 02:06:38 2022 +0000

    Removed hacky line.

commit 5b23eb7
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Mar 20 01:44:57 2022 +0000

    Added support for Parameters.
    - Supporting int, str, bool, float, JSONType

commit c9378e9
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sun Mar 20 00:14:10 2022 +0000

    Removed todos + added some validation logic.

commit 7250a44
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Mar 19 23:45:15 2022 +0000

    Fixing logs related change from master.

commit d125978
Merge: 8cdac53 7e210a2
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Mar 19 23:42:48 2022 +0000

    Merge branch 'master' into aft-mm

commit 8cdac53
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Mar 19 23:36:47 2022 +0000

    making changes sync with master

commit 5a93d9f
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Mar 19 23:29:47 2022 +0000

    Fixed bug when using catch + retry

commit 62bc8df
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Mar 19 22:58:37 2022 +0000

    Changed retry setting.

commit 563a200
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Mar 19 22:42:57 2022 +0000

    Fixed setting `task_id` :
    - switch task-id from airflow job is to hash to "runid/stepname"
    - refactor xcom setting variables
    - added comments

commit e2a1e50
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Mar 19 17:51:59 2022 +0000

    setting retry logic.

commit a697b56
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Mar 17 01:02:11 2022 +0000

    Nit fix.

commit 68f13be
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Mar 16 20:46:19 2022 +0000

    Added @schedule support + readme

commit 57bdde5
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Tue Mar 15 19:47:06 2022 +0000

    Fixed setting run-id / task-id to labels in k8s
    - Fixed setting run-id has from cli macro
    - added hashing macro to ensure that jinja template set the correct run-id to k8s labels
    -

commit 3d6c319
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Tue Mar 15 05:39:04 2022 +0000

    Got linear workflows working on airflow.
    - Still not feature complete as lots of args are still unfilled / lots of unknows
    - minor tweek in eks to ensure airflow is k8s compatible.
    - passing state around via xcom-push
    - HACK : AWS keys are passed in a shady way. : Reverse this soon.

commit db074b8
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Fri Mar 11 12:34:33 2022 -0800

    Tweeks

commit a9f0468
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Tue Mar 1 17:14:47 2022 -0800

    some changes based on savin's comments.
    - Added changes to task datastore for different reason : (todo) Decouple these
    - Added comments to SFN for reference.
    - Airflow DAG is no longer dependent on metaflow

commit f32d089
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Wed Feb 23 00:54:17 2022 -0800

    First version of dynamic dag compiler.
    - Not completely finished code
    - Creates generic .py file a JSON that is parsed to create Airflow DAG.
    - Currently only boiler plate to make a linear dag but doesn't execute anything.
    -  Unfinished code.

commit d2def66
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Sat Feb 19 14:01:47 2022 -0800

    more tweeks.

commit b176311
Author: Valay Dave <valaygaurang@gmail.com>
Date:   Thu Feb 17 09:04:29 2022 -0800

    commit 0
    - unfinished code.

* Making version compatibility changes.
- Minimum support version to 2.2.0

* Task-id macro related logic refactor
- Done for better version support.

* bug fix: param related task-id setting.

* applied black

* Reverting `decorators.py` to master

* Move sys.path insert earlier in s3op.py (#1098)

* Update setup.py (#1099)

* Fix for env_escape bug when importing local packages (#1100)

* Black reformat

* Bump to 2.7.5 (#1102)

* Fix another issue with the escape hatch and paths (#1105)

* Bump to 2.7.6 (#1106)

* add a flag to overwrite config when running metaflow configure sandbox (#1103)

* Azure Storage

* fix BrokenProcessPool import for older pythons

* Fix batch

* add azure configure, cannot validate storage account url after all (catch-22 in configure)

* rename storage account url to AZURE_STORAGE_BLOB_SERVICE_ENDPOINT

* fix indent

* clean up kubernetes config cli, add secrets

* remove signature

* Fix issue with get_pinned_conda_libs and metaflow extensions

* clean up includefile a bit, remove from_env (not used)

* Black reformat

* Fix order of imports in __init__.py

* More black

* fix "configure azure" merge conflict

Co-authored-by: Oleg Avdeev <oleg.v.avdeev@gmail.com>
Co-authored-by: Ville Tuulos <tuulos@gmail.com>
Co-authored-by: Romain Cledat <rcledat@netflix.com>
Co-authored-by: Romain <romain-intel@users.noreply.github.com>
Co-authored-by: Valay Dave <valaygaurang@gmail.com>
Co-authored-by: Savin <savingoyal@gmail.com>
Co-authored-by: Shashank Srikanth <108034001+hunsdiecker@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

8 participants