Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

apache-airflow: Export PYTHONPATH in wrapper for gunicorn processes #87620

Merged
merged 1 commit into from May 12, 2020

Conversation

@jonringer
Copy link
Contributor

jonringer commented May 11, 2020

Motivation for this change

closes: #84512 #87612

$ ./result/bin/airflow webserver
[2020-05-11 13:12:13,168] {__init__.py:51} INFO - Using executor SequentialExecutor
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[2020-05-11 13:12:13,524] {dagbag.py:90} INFO - Filling up the DagBag from /home/jon/airflow/dags
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
=================================================================
[2020-05-11 13:12:14 -0700] [21413] [INFO] Starting gunicorn 20.0.4
[2020-05-11 13:12:14 -0700] [21413] [INFO] Listening at: http://0.0.0.0:8080 (21413)
[2020-05-11 13:12:14 -0700] [21413] [INFO] Using worker: sync
[2020-05-11 13:12:14 -0700] [21416] [INFO] Booting worker with pid: 21416
[2020-05-11 13:12:14 -0700] [21417] [INFO] Booting worker with pid: 21417
[2020-05-11 13:12:14 -0700] [21418] [INFO] Booting worker with pid: 21418
[2020-05-11 13:12:14,432] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-05-11 13:12:14,440] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-05-11 13:12:14,464] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-05-11 13:12:14 -0700] [21419] [INFO] Booting worker with pid: 21419
[2020-05-11 13:12:14,553] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-05-11 13:12:14,654] {dagbag.py:90} INFO - Filling up the DagBag from /home/jon/airflow/dags
[2020-05-11 13:12:14,656] {dagbag.py:90} INFO - Filling up the DagBag from /home/jon/airflow/dags
[2020-05-11 13:12:14,688] {dagbag.py:90} INFO - Filling up the DagBag from /home/jon/airflow/dags
[2020-05-11 13:12:14,773] {dagbag.py:90} INFO - Filling up the DagBag from /home/jon/airflow/dags
[2020-05-11 13:12:45 -0700] [21413] [INFO] Handling signal: ttin
[2020-05-11 13:12:45 -0700] [21429] [INFO] Booting worker with pid: 21429
[2020-05-11 13:12:45,676] {__init__.py:51} INFO - Using executor SequentialExecutor
[2020-05-11 13:12:45,885] {dagbag.py:90} INFO - Filling up the DagBag from /home/jon/airflow/dags
Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

cc @GTrunSec @glittershark

@bhipple
Copy link
Contributor

bhipple commented May 12, 2020

Result of nixpkgs-review pr 87620 1

1 package failed to build:
- python38Packages.apache-airflow
1 package built:
- apache-airflow (python37Packages.apache-airflow)
@GTrunSec
Copy link
Contributor

GTrunSec commented May 12, 2020

Result of nixpkgs-review pr 87620 1

1 package failed to build:

  • python38Packages.apache-airflow
    1 package built:

The airflow does not support python 3.8

official ref: Both versions are currently incompatible with Python 3.8 due to a known compatibility issue with a dependent library

Also, we still need to solve some dependent libraries. https://github.com/apache/airflow/blob/master/requirements/requirements-python3.7.txt

Copy link
Contributor

glittershark left a comment

This fixes the issue for me!

@jonringer
Copy link
Contributor Author

jonringer commented May 12, 2020

looks like this solves the issue without double-wrapping.

@jonringer jonringer merged commit e240c5d into NixOS:master May 12, 2020
17 checks passed
17 checks passed
apache-airflow, apache-airflow.passthru.tests on x86_64-darwin Failure
Details
Evaluation Performance Report Evaluator Performance Report
Details
apache-airflow, apache-airflow.passthru.tests on aarch64-linux Success
Details
apache-airflow, apache-airflow.passthru.tests on x86_64-linux Success
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="e126dfa"; rev="e126dfa629d3bebcd60bff3394a65b98401fefca"; } ./pkgs/t
Details
grahamcofborg-eval-lib-tests nix-build --arg pkgs import ./. {} ./lib/tests/release.nix
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="e126dfa"; rev="e126dfa629d3bebcd60bff3394a65b98401fefca"; } ./nixos/
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="e126dfa"; rev="e126dfa629d3bebcd60bff3394a65b98401fefca"; } ./nixos/
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="e126dfa"; rev="e126dfa629d3bebcd60bff3394a65b98401fefca"; } ./nixos/
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="e126dfa"; rev="e126dfa629d3bebcd60bff3394a65b98401fefca"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="e126dfa"; rev="e126dfa629d3bebcd60bff3394a65b98401fefca"; } ./pkgs/t
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="e126dfa"; rev="e126dfa629d3bebcd60bff3394a65b98401fefca"; } ./pkgs/t
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

4 participants
You can’t perform that action at this time.