Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-6157] Support for multiple executors #6725

Closed

Conversation

@potiuk
Copy link
Member

potiuk commented Dec 4, 2019

Make sure you have checked all steps below.

Jira

  • My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
    • https://issues.apache.org/jira/browse/AIRFLOW-6157
    • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.
    • In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal (AIP).
    • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Description

  • Here are some details about my PR, including screenshots of any UI changes:

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain docstrings that explain what it does
    • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release
Copy link
Contributor

nuclearpinguin left a comment

I really curious about this proposal!

from airflow.executors.kubernetes_executor import KubernetesExecutor
return KubernetesExecutor()
return cls.get_or_create_executor(executor_name=executor_name,
create_executor=ExecutorLoader.create_kubernetes_executor)

This comment has been minimized.

Copy link
@nuclearpinguin

nuclearpinguin Dec 4, 2019

Contributor

I wonder if we can use some mapping here because pylint is going to complain about to much return statements when we add one more executor.

def _get_executor(cls, executor_name: str) -> BaseExecutor:
    mapping = {
        ExecutorLoader.LOCAL_EXECUTOR: ExecutorLoader.create_local_executor
        ...
    }
    if executor_name in mapping:
        method = mapping[executor_name]
        return cls.get_or_create_executor(executor_name, method)
    else:
        ...

This comment has been minimized.

Copy link
@potiuk

potiuk Dec 4, 2019

Author Member

sure. We can set variable and make single return :)


def sync(self) -> None:
for executor in self.executor_set:
executor.sync()

This comment has been minimized.

Copy link
@nuclearpinguin

nuclearpinguin Dec 4, 2019

Contributor

I just wonder if we should use try / except in such loops? Because now when first executor fails everything fails. But using exception handling we can "execute as much as possible" and then throw an exception. WDYT?

This comment has been minimized.

Copy link
@potiuk

potiuk Dec 4, 2019

Author Member

Yes. That's a great comment.

@potiuk

This comment has been minimized.

Copy link
Member Author

potiuk commented Dec 9, 2019

Multiple executors are not needed (we get rid of KNative executor) but some of the PR will be opened in a moment (I will retain the JIRA id)

@potiuk potiuk closed this Dec 9, 2019
@nuclearpinguin

This comment has been minimized.

Copy link
Contributor

nuclearpinguin commented Dec 9, 2019

@potiuk I just wonder about simplifying types. Can we make a static "tasks run" command builder so the ['airflow', 'tasks', ...] type can be abandoned? WDYT?

@potiuk

This comment has been minimized.

Copy link
Member Author

potiuk commented Dec 9, 2019

@potiuk I just wonder about simplifying types. Can we make a static "tasks run" command builder so the ['airflow', 'tasks', ...] type can be abandoned? WDYT?

I guess this is a discussion for another PR ?

@mik-laj

This comment has been minimized.

Copy link
Member

mik-laj commented Dec 9, 2019

@nuclearpinguin It looks like one of the tasks that we can do for our second client. I think, however, that the executor should not be based on sending commands, but objects that describe the launch, and only the worker should build the appropriate command.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.