Skip to content

Comments

Attempt to seperate JobWatcher from multiprocessing#11069

Closed
dimberman wants to merge 1 commit intoapache:masterfrom
astronomer:seperate-job-watcher-again
Closed

Attempt to seperate JobWatcher from multiprocessing#11069
dimberman wants to merge 1 commit intoapache:masterfrom
astronomer:seperate-job-watcher-again

Conversation

@dimberman
Copy link
Contributor


^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in UPDATING.md.

@boring-cyborg boring-cyborg bot added area:Scheduler including HA (high availability) scheduler k8s labels Sep 21, 2020
@potiuk
Copy link
Member

potiuk commented Sep 22, 2020

Looking at the errors/timeouts. I think we still drag too many things with the run method. I am still figuring out how exactly multiprocessing.Process pickles it's run parameter. Seems that making the method static did not help. @ashb - I'd love to understand how it works. Maybe you can explain what happens here? This is very interesting, but looks like we are trying to send half ot Airflow objects between the processes with that single run= method.

BTW. Did I tell too many times, that I hate multiprocessing?

@stale
Copy link

stale bot commented Nov 15, 2020

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Nov 15, 2020
@kaxil kaxil added provider:cncf-kubernetes Kubernetes (k8s) provider related issues and removed area:k8s labels Nov 18, 2020
@stale stale bot removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Nov 18, 2020
@dimberman dimberman closed this Dec 14, 2020
@dimberman dimberman deleted the seperate-job-watcher-again branch December 14, 2020 16:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:Scheduler including HA (high availability) scheduler provider:cncf-kubernetes Kubernetes (k8s) provider related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants