Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSS Gate Workshop: MonotaRO: 2019-12-02: sheng_wei: Airflow: Work log #1358

Closed
seii-saintway opened this issue Dec 2, 2019 · 9 comments
Closed

Comments

@seii-saintway
Copy link
Contributor

@seii-saintway seii-saintway commented Dec 2, 2019

This is a work log of a "OSS Gate workshop".
"OSS Gate workshop" is an activity to increase OSS developers.
Here's been discussed in Japanese. Thanks.

作業ログ作成時の説明

以下のテンプレートを埋めてタイトルに設定します。埋め方例はスクロールすると見えてきます。

OSS Gate Workshop: ${LOCATION}: ${YEAR}-${MONTH}-${DAY}: ${ACCOUNT_NAME}: ${OSS_NAME}: Work log

タイトル例↓:

OSS Gate Workshop: Tokyo: 2017-01-16: kou: Rabbit: Work log

OSS Gateワークショップ関連情報

  • スライド:ワークショップの進行に使っているスライドがあります。
  • チャット:OSS開発に関することならなんでも相談できます。ワークショップが終わった後もオンラインで相談しながら継続的にOSSの開発に参加しましょう!
  • シナリオ:ワークショップの目的・内容・進め方の詳細が書いています。
  • 過去のビギナーの作業ログ:他の人の作業ログから学べることがいろいろあるはずです。
@seii-saintway

This comment has been minimized.

Copy link
Contributor Author

@seii-saintway seii-saintway commented Dec 2, 2019

@seii-saintway

This comment has been minimized.

Copy link
Contributor Author

@seii-saintway seii-saintway commented Dec 2, 2019

%%bash
# airflow needs a home, ~/airflow is the default,
# but you can lay foundation somewhere else if you prefer
# (optional)
mkdir ~/airflow
%env AIRFLOW_HOME=~/airflow
@seii-saintway

This comment has been minimized.

Copy link
Contributor Author

@seii-saintway seii-saintway commented Dec 2, 2019

!sudo -HE pip install marshmallow==2.18.0
Collecting marshmallow==2.18.0

[ Downloading https://files.pythonhosted.org/packages/9a/3c/4cc463c53136dc503f9ee234c4a6592e1c7411cb362f844e80df70361f29/marshmallow-2.18.0-py2.py3-none-any.whl (50kB)
[ |████████████████████████████████| 51kB 15.9MB/s eta 0:00:01
[Installing collected packages: marshmallow
Found existing installation: marshmallow 3.2.2
Uninstalling marshmallow-3.2.2:
Successfully uninstalled marshmallow-3.2.2
Successfully installed marshmallow-2.18.0

%%bash
# install from pypi using pip
sudo -HE pip install apache-airflow
Requirement already satisfied: apache-airflow in /opt/tljh/user/lib/python3.6/site-packages (1.10.6)
Requirement already satisfied: cached-property~=1.5 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.5.1)
Requirement already satisfied: future<0.17,>=0.16.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.16.0)
Requirement already satisfied: text-unidecode==1.2 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.2)
Requirement already satisfied: unicodecsv>=0.14.1 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.14.1)
Requirement already satisfied: iso8601>=0.1.12 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.1.12)
Requirement already satisfied: croniter<0.4,>=0.3.17 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.3.30)
Requirement already satisfied: flask-login<0.5,>=0.3 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.4.1)
Requirement already satisfied: python-dateutil<3,>=2.3 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (2.8.0)
Requirement already satisfied: requests<3,>=2.20.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (2.22.0)
Requirement already satisfied: colorlog==4.0.2 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (4.0.2)
Requirement already satisfied: dill<0.4,>=0.2.2 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.3.1.1)
Requirement already satisfied: python-daemon<2.2,>=2.1.1 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (2.1.2)
Requirement already satisfied: marshmallow-sqlalchemy<0.19.0,>=0.16.1 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.18.0)
Requirement already satisfied: psutil<6.0.0,>=4.2.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (5.6.3)
Requirement already satisfied: termcolor==1.1.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.1.0)
Requirement already satisfied: flask-admin==1.5.3 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.5.3)
Requirement already satisfied: flask-swagger==0.2.13 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.2.13)
Requirement already satisfied: pygments<3.0,>=2.0.1 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (2.4.2)
Requirement already satisfied: sqlalchemy~=1.3 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.3.8)
Requirement already satisfied: funcsigs==1.0.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.0.0)
Requirement already satisfied: flask<2.0,>=1.1.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.1.1)
Requirement already satisfied: tenacity==4.12.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (4.12.0)
Requirement already satisfied: setproctitle<2,>=1.1.8 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.1.10)
Requirement already satisfied: thrift>=0.9.2 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.13.0)
Requirement already satisfied: pendulum==1.4.4 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.4.4)
Requirement already satisfied: jinja2<2.11.0,>=2.10.1 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (2.10.1)
Requirement already satisfied: gunicorn<20.0,>=19.5.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (19.9.0)
Requirement already satisfied: argcomplete~=1.10 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.10.3)
Requirement already satisfied: flask-appbuilder<2.0.0,>=1.12.5 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.13.1)
Requirement already satisfied: lazy-object-proxy~=1.3 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.4.3)
Requirement already satisfied: json-merge-patch==0.2 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.2)
Requirement already satisfied: flask-wtf<0.15,>=0.14.2 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.14.2)
Requirement already satisfied: tabulate<0.9,>=0.7.5 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.8.6)
Requirement already satisfied: flask-caching<1.4.0,>=1.3.3 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.3.3)
Requirement already satisfied: configparser<3.6.0,>=3.5.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (3.5.3)
Requirement already satisfied: zope.deprecation<5.0,>=4.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (4.4.0)
Requirement already satisfied: markdown<3.0,>=2.5.2 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (2.6.11)
Requirement already satisfied: pandas<1.0.0,>=0.17.1 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.25.1)
Requirement already satisfied: alembic<2.0,>=1.0 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.1.0)
Requirement already satisfied: tzlocal<2.0.0,>=1.4 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (1.5.1)
Requirement already satisfied: graphviz>=0.12 in /opt/tljh/user/lib/python3.6/site-packages (from apache-airflow) (0.13.2)
Requirement already satisfied: six>=1.5 in /opt/tljh/user/lib/python3.6/site-packages (from python-dateutil<3,>=2.3->apache-airflow) (1.12.0)
Requirement already satisfied: certifi>=2017.4.17 in /opt/tljh/user/lib/python3.6/site-packages (from requests<3,>=2.20.0->apache-airflow) (2019.9.11)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/tljh/user/lib/python3.6/site-packages (from requests<3,>=2.20.0->apache-airflow) (1.25.3)
Requirement already satisfied: idna<2.9,>=2.5 in /opt/tljh/user/lib/python3.6/site-packages (from requests<3,>=2.20.0->apache-airflow) (2.8)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/tljh/user/lib/python3.6/site-packages (from requests<3,>=2.20.0->apache-airflow) (3.0.4)
Requirement already satisfied: setuptools in /opt/tljh/user/lib/python3.6/site-packages (from python-daemon<2.2,>=2.1.1->apache-airflow) (41.4.0)
Requirement already satisfied: docutils in /opt/tljh/user/lib/python3.6/site-packages (from python-daemon<2.2,>=2.1.1->apache-airflow) (0.15.2)
Requirement already satisfied: lockfile>=0.10 in /opt/tljh/user/lib/python3.6/site-packages (from python-daemon<2.2,>=2.1.1->apache-airflow) (0.12.2)
Requirement already satisfied: marshmallow>=2.0.0 in /opt/tljh/user/lib/python3.6/site-packages (from marshmallow-sqlalchemy<0.19.0,>=0.16.1->apache-airflow) (2.18.0)
Requirement already satisfied: wtforms in /opt/tljh/user/lib/python3.6/site-packages (from flask-admin==1.5.3->apache-airflow) (2.2.1)
Requirement already satisfied: PyYAML>=3.0 in /opt/tljh/user/lib/python3.6/site-packages (from flask-swagger==0.2.13->apache-airflow) (5.1.2)
Requirement already satisfied: ordereddict in /opt/tljh/user/lib/python3.6/site-packages (from funcsigs==1.0.0->apache-airflow) (1.1)
Requirement already satisfied: itsdangerous>=0.24 in /opt/tljh/user/lib/python3.6/site-packages (from flask<2.0,>=1.1.0->apache-airflow) (1.1.0)
Requirement already satisfied: Werkzeug>=0.15 in /opt/tljh/user/lib/python3.6/site-packages (from flask<2.0,>=1.1.0->apache-airflow) (0.16.0)
Requirement already satisfied: click>=5.1 in /opt/tljh/user/lib/python3.6/site-packages (from flask<2.0,>=1.1.0->apache-airflow) (7.0)
Requirement already satisfied: pytzdata>=2018.3.0.0 in /opt/tljh/user/lib/python3.6/site-packages (from pendulum==1.4.4->apache-airflow) (2019.3)
Requirement already satisfied: MarkupSafe>=0.23 in /opt/tljh/user/lib/python3.6/site-packages (from jinja2<2.11.0,>=2.10.1->apache-airflow) (1.1.1)
Requirement already satisfied: apispec[yaml]>=1.1.1<2 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (3.1.0)
Requirement already satisfied: Flask-OpenID<2,>=1.2.5 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (1.2.5)
Requirement already satisfied: marshmallow-enum<2,>=1.4.1 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (1.5.1)
Requirement already satisfied: colorama<1,>=0.3.9 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (0.4.1)
Requirement already satisfied: Flask-JWT-Extended<4,>=3.18 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (3.24.1)
Requirement already satisfied: prison==0.1.0 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (0.1.0)
Requirement already satisfied: Flask-Babel<1,>=0.11.1 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (0.12.2)
Requirement already satisfied: jsonschema>=3.0.1<4 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (3.0.2)
Requirement already satisfied: PyJWT>=1.7.1 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (1.7.1)
Requirement already satisfied: Flask-SQLAlchemy<3,>=2.3 in /opt/tljh/user/lib/python3.6/site-packages (from flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (2.4.1)
Requirement already satisfied: numpy>=1.13.3 in /opt/tljh/user/lib/python3.6/site-packages (from pandas<1.0.0,>=0.17.1->apache-airflow) (1.17.2)
Requirement already satisfied: pytz>=2017.2 in /opt/tljh/user/lib/python3.6/site-packages (from pandas<1.0.0,>=0.17.1->apache-airflow) (2019.2)
Requirement already satisfied: Mako in /opt/tljh/user/lib/python3.6/site-packages (from alembic<2.0,>=1.0->apache-airflow) (1.1.0)
Requirement already satisfied: python-editor>=0.3 in /opt/tljh/user/lib/python3.6/site-packages (from alembic<2.0,>=1.0->apache-airflow) (1.0.4)
Requirement already satisfied: python3-openid>=2.0 in /opt/tljh/user/lib/python3.6/site-packages (from Flask-OpenID<2,>=1.2.5->flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (3.1.0)
Requirement already satisfied: Babel>=2.3 in /opt/tljh/user/lib/python3.6/site-packages (from Flask-Babel<1,>=0.11.1->flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (2.7.0)
Requirement already satisfied: attrs>=17.4.0 in /opt/tljh/user/lib/python3.6/site-packages (from jsonschema>=3.0.1<4->flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (19.1.0)
Requirement already satisfied: pyrsistent>=0.14.0 in /opt/tljh/user/lib/python3.6/site-packages (from jsonschema>=3.0.1<4->flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (0.15.4)
Requirement already satisfied: defusedxml in /opt/tljh/user/lib/python3.6/site-packages (from python3-openid>=2.0->Flask-OpenID<2,>=1.2.5->flask-appbuilder<2.0.0,>=1.12.5->apache-airflow) (0.6.0)
@seii-saintway

This comment has been minimized.

Copy link
Contributor Author

@seii-saintway seii-saintway commented Dec 2, 2019

%%bash
# initialize the database
airflow initdb
DB: sqlite:////home/jupyter-sheng_wei/airflow/airflow.db
[2019-12-02 14:25:56,971] {db.py:368} INFO - Creating tables
Done.


INFO  [alembic.runtime.migration] Context impl SQLiteImpl.
INFO  [alembic.runtime.migration] Will assume non-transactional DDL.
INFO  [alembic.runtime.migration] Running upgrade  -> e3a246e0dc1, current schema
INFO  [alembic.runtime.migration] Running upgrade e3a246e0dc1 -> 1507a7289a2f, create is_encrypted
/opt/tljh/user/lib/python3.6/site-packages/alembic/ddl/sqlite.py:39: UserWarning: Skipping unsupported ALTER for creation of implicit constraint
  "Skipping unsupported ALTER for "
INFO  [alembic.runtime.migration] Running upgrade 1507a7289a2f -> 13eb55f81627, maintain history for compatibility with earlier migrations
INFO  [alembic.runtime.migration] Running upgrade 13eb55f81627 -> 338e90f54d61, More logging into task_instance
INFO  [alembic.runtime.migration] Running upgrade 338e90f54d61 -> 52d714495f0, job_id indices
INFO  [alembic.runtime.migration] Running upgrade 52d714495f0 -> 502898887f84, Adding extra to Log
INFO  [alembic.runtime.migration] Running upgrade 502898887f84 -> 1b38cef5b76e, add dagrun
INFO  [alembic.runtime.migration] Running upgrade 1b38cef5b76e -> 2e541a1dcfed, task_duration
INFO  [alembic.runtime.migration] Running upgrade 2e541a1dcfed -> 40e67319e3a9, dagrun_config
INFO  [alembic.runtime.migration] Running upgrade 40e67319e3a9 -> 561833c1c74b, add password column to user
INFO  [alembic.runtime.migration] Running upgrade 561833c1c74b -> 4446e08588, dagrun start end
INFO  [alembic.runtime.migration] Running upgrade 4446e08588 -> bbc73705a13e, Add notification_sent column to sla_miss
INFO  [alembic.runtime.migration] Running upgrade bbc73705a13e -> bba5a7cfc896, Add a column to track the encryption state of the 'Extra' field in connection
INFO  [alembic.runtime.migration] Running upgrade bba5a7cfc896 -> 1968acfc09e3, add is_encrypted column to variable table
INFO  [alembic.runtime.migration] Running upgrade 1968acfc09e3 -> 2e82aab8ef20, rename user table
INFO  [alembic.runtime.migration] Running upgrade 2e82aab8ef20 -> 211e584da130, add TI state index
INFO  [alembic.runtime.migration] Running upgrade 211e584da130 -> 64de9cddf6c9, add task fails journal table
INFO  [alembic.runtime.migration] Running upgrade 64de9cddf6c9 -> f2ca10b85618, add dag_stats table
INFO  [alembic.runtime.migration] Running upgrade f2ca10b85618 -> 4addfa1236f1, Add fractional seconds to mysql tables
INFO  [alembic.runtime.migration] Running upgrade 4addfa1236f1 -> 8504051e801b, xcom dag task indices
INFO  [alembic.runtime.migration] Running upgrade 8504051e801b -> 5e7d17757c7a, add pid field to TaskInstance
INFO  [alembic.runtime.migration] Running upgrade 5e7d17757c7a -> 127d2bf2dfa7, Add dag_id/state index on dag_run table
INFO  [alembic.runtime.migration] Running upgrade 127d2bf2dfa7 -> cc1e65623dc7, add max tries column to task instance
INFO  [alembic.runtime.migration] Running upgrade cc1e65623dc7 -> bdaa763e6c56, Make xcom value column a large binary
INFO  [alembic.runtime.migration] Running upgrade bdaa763e6c56 -> 947454bf1dff, add ti job_id index
INFO  [alembic.runtime.migration] Running upgrade 947454bf1dff -> d2ae31099d61, Increase text size for MySQL (not relevant for other DBs' text types)
INFO  [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 0e2a74e0fc9f, Add time zone awareness
INFO  [alembic.runtime.migration] Running upgrade d2ae31099d61 -> 33ae817a1ff4, kubernetes_resource_checkpointing
INFO  [alembic.runtime.migration] Running upgrade 33ae817a1ff4 -> 27c6a30d7c24, kubernetes_resource_checkpointing
INFO  [alembic.runtime.migration] Running upgrade 27c6a30d7c24 -> 86770d1215c0, add kubernetes scheduler uniqueness
INFO  [alembic.runtime.migration] Running upgrade 86770d1215c0, 0e2a74e0fc9f -> 05f30312d566, merge heads
INFO  [alembic.runtime.migration] Running upgrade 05f30312d566 -> f23433877c24, fix mysql not null constraint
INFO  [alembic.runtime.migration] Running upgrade f23433877c24 -> 856955da8476, fix sqlite foreign key
INFO  [alembic.runtime.migration] Running upgrade 856955da8476 -> 9635ae0956e7, index-faskfail
INFO  [alembic.runtime.migration] Running upgrade 9635ae0956e7 -> dd25f486b8ea, add idx_log_dag
INFO  [alembic.runtime.migration] Running upgrade dd25f486b8ea -> bf00311e1990, add index to taskinstance
INFO  [alembic.runtime.migration] Running upgrade 9635ae0956e7 -> 0a2a5b66e19d, add task_reschedule table
INFO  [alembic.runtime.migration] Running upgrade 0a2a5b66e19d, bf00311e1990 -> 03bc53e68815, merge_heads_2
INFO  [alembic.runtime.migration] Running upgrade 03bc53e68815 -> 41f5f12752f8, add superuser field
INFO  [alembic.runtime.migration] Running upgrade 41f5f12752f8 -> c8ffec048a3b, add fields to dag
INFO  [alembic.runtime.migration] Running upgrade c8ffec048a3b -> dd4ecb8fbee3, Add schedule interval to dag
INFO  [alembic.runtime.migration] Running upgrade dd4ecb8fbee3 -> 939bb1e647c8, task reschedule fk on cascade delete
INFO  [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 6e96a59344a4, Make TaskInstance.pool not nullable
INFO  [alembic.runtime.migration] Running upgrade 6e96a59344a4 -> 74effc47d867, change datetime to datetime2(6) on MSSQL tables
INFO  [alembic.runtime.migration] Running upgrade c8ffec048a3b -> a56c9515abdc, Remove dag_stat table
INFO  [alembic.runtime.migration] Running upgrade 939bb1e647c8 -> 004c1210f153, increase queue name size limit
@seii-saintway

This comment has been minimized.

Copy link
Contributor Author

@seii-saintway seii-saintway commented Dec 2, 2019

%%bash
# start the web server, default port is 8080
airflow webserver -p 8080
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[[34m2019-12-02 14:30:48,749[0m] {[34m__init__.py:[0m51} INFO[0m - Using executor [1mSequentialExecutor[0m[0m
[[34m2019-12-02 14:30:48,750[0m] {[34mdagbag.py:[0m92} INFO[0m - Filling up the DagBag from [1m/home/jupyter-sheng_wei/airflow/dags[0m[0m
Running the Gunicorn Server with:
Workers: 4 sync
Host: 0.0.0.0:8080
Timeout: 120
Logfiles: - -
=================================================================            
[2019-12-02 14:30:49 +0900] [14677] [INFO] Starting gunicorn 19.9.0
[2019-12-02 14:30:49 +0900] [14677] [INFO] Listening at: http://0.0.0.0:8080 (14677)
[2019-12-02 14:30:49 +0900] [14677] [INFO] Using worker: sync
[2019-12-02 14:30:49 +0900] [14682] [INFO] Booting worker with pid: 14682
[2019-12-02 14:30:50 +0900] [14683] [INFO] Booting worker with pid: 14683
[2019-12-02 14:30:50 +0900] [14684] [INFO] Booting worker with pid: 14684
[2019-12-02 14:30:50 +0900] [14685] [INFO] Booting worker with pid: 14685
[[34m2019-12-02 14:30:50,996[0m] {[34m__init__.py:[0m51} INFO[0m - Using executor [1mSequentialExecutor[0m[0m
[[34m2019-12-02 14:30:51,008[0m] {[34mdagbag.py:[0m92} INFO[0m - Filling up the DagBag from [1m/home/jupyter-sheng_wei/airflow/dags[0m[0m
[[34m2019-12-02 14:30:51,289[0m] {[34m__init__.py:[0m51} INFO[0m - Using executor [1mSequentialExecutor[0m[0m
[[34m2019-12-02 14:30:51,304[0m] {[34mdagbag.py:[0m92} INFO[0m - Filling up the DagBag from [1m/home/jupyter-sheng_wei/airflow/dags[0m[0m
[[34m2019-12-02 14:30:51,442[0m] {[34m__init__.py:[0m51} INFO[0m - Using executor [1mSequentialExecutor[0m[0m
[[34m2019-12-02 14:30:51,456[0m] {[34mdagbag.py:[0m92} INFO[0m - Filling up the DagBag from [1m/home/jupyter-sheng_wei/airflow/dags[0m[0m
[[34m2019-12-02 14:30:51,469[0m] {[34m__init__.py:[0m51} INFO[0m - Using executor [1mSequentialExecutor[0m[0m
[[34m2019-12-02 14:30:51,480[0m] {[34mdagbag.py:[0m92} INFO[0m - Filling up the DagBag from [1m/home/jupyter-sheng_wei/airflow/dags[0m[0m
123.226.242.94 - - [02/Dec/2019:14:31:07 +0900] "GET / HTTP/1.1" 302 221 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
123.226.242.94 - - [02/Dec/2019:14:31:08 +0900] "GET /admin/ HTTP/1.1" 200 115313 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
123.226.242.94 - - [02/Dec/2019:14:31:08 +0900] "GET /admin/airflow/blocked HTTP/1.1" 200 2 "http://jhub.name:8080/admin/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
123.226.242.94 - - [02/Dec/2019:14:31:08 +0900] "GET /admin/airflow/dag_stats HTTP/1.1" 200 9486 "http://jhub.name:8080/admin/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
123.226.242.94 - - [02/Dec/2019:14:31:08 +0900] "GET /admin/airflow/task_stats HTTP/1.1" 200 30510 "http://jhub.name:8080/admin/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.97 Safari/537.36"
[2019-12-02 14:31:23 +0900] [14677] [INFO] Handling signal: ttin
[2019-12-02 14:31:23 +0900] [14687] [INFO] Booting worker with pid: 14687
[[34m2019-12-02 14:31:23,855[0m] {[34m__init__.py:[0m51} INFO[0m - Using executor [1mSequentialExecutor[0m[0m
[[34m2019-12-02 14:31:23,856[0m] {[34mdagbag.py:[0m92} INFO[0m - Filling up the DagBag from [1m/home/jupyter-sheng_wei/airflow/dags[0m[0m
[2019-12-02 14:31:24 +0900] [14677] [INFO] Handling signal: ttou
[2019-12-02 14:31:24 +0900] [14682] [INFO] Worker exiting (pid: 14682)
[0m[2019-12-02 14:31:30 +0900] [14677] [INFO] Handling signal: int
[2019-12-02 14:31:30 +0900] [14683] [INFO] Worker exiting (pid: 14683)
[2019-12-02 14:31:30 +0900] [14684] [INFO] Worker exiting (pid: 14684)
[0m[0m[2019-12-02 14:31:30 +0900] [14685] [INFO] Worker exiting (pid: 14685)
[0m[2019-12-02 14:31:30 +0900] [14687] [INFO] Worker exiting (pid: 14687)
[0m[2019-12-02 14:31:31 +0900] [14677] [INFO] Shutting down: Master
[0m[0m
@seii-saintway

This comment has been minimized.

Copy link
Contributor Author

@seii-saintway seii-saintway commented Dec 2, 2019

# start the scheduler
airflow scheduler
  ____________       _____________
 ____    |__( )_________  __/__  /________      __
____  /| |_  /__  ___/_  /_ __  /_  __ \_ | /| / /
___  ___ |  / _  /   _  __/ _  / / /_/ /_ |/ |/ /
 _/_/  |_/_/  /_/    /_/    /_/  \____/____/|__/
[[34m2019-12-02 14:33:56,327[0m] {[34m__init__.py:[0m51} INFO[0m - Using executor [1mSequentialExecutor[0m[0m
[[34m2019-12-02 14:33:56,343[0m] {[34mscheduler_job.py:[0m1320} INFO[0m - Starting the scheduler[0m
[[34m2019-12-02 14:33:56,343[0m] {[34mscheduler_job.py:[0m1328} INFO[0m - Running execute loop for -1 seconds[0m
[[34m2019-12-02 14:33:56,345[0m] {[34mscheduler_job.py:[0m1329} INFO[0m - Processing each file at most -1 times[0m
[[34m2019-12-02 14:33:56,346[0m] {[34mscheduler_job.py:[0m1332} INFO[0m - Searching for files in [1m/home/jupyter-sheng_wei/airflow/dags[0m[0m
[[34m2019-12-02 14:33:56,351[0m] {[34mscheduler_job.py:[0m1334} INFO[0m - There are 20 files in [1m/home/jupyter-sheng_wei/airflow/dags[0m[0m
[[34m2019-12-02 14:33:56,354[0m] {[34mscheduler_job.py:[0m1382} INFO[0m - Resetting orphaned tasks for active dag runs[0m
[[34m2019-12-02 14:33:56,366[0m] {[34mdag_processing.py:[0m556} INFO[0m - Launched DagFileProcessorManager with pid: 14781[0m
[[34m2019-12-02 14:33:56,374[0m] {[34msettings.py:[0m54} INFO[0m - Configured default timezone <Timezone [UTC]>[0m
[2019-12-02 14:33:56,384] {dag_processing.py:760} ERROR - Cannot use more than 1 thread when using sqlite. Setting parallelism to 1
[[34m2019-12-02 14:34:11,631[0m] {[34mhelpers.py:[0m308} INFO[0m - Sending Signals.SIGTERM to GPID 14781[0m
[[34m2019-12-02 14:34:11,636[0m] {[34mhelpers.py:[0m286} INFO[0m - Process [1mpsutil.Process(pid=14781, status='terminated')[0m (14781) terminated with exit code 0[0m
[[34m2019-12-02 14:34:11,637[0m] {[34mscheduler_job.py:[0m1361} INFO[0m - Exited execute loop[0m
[0m
@seii-saintway

This comment has been minimized.

Copy link
Contributor Author

@seii-saintway seii-saintway commented Dec 2, 2019

↑実際には、エラーでしたが、止まりませんでした。
自動的に、max_threads を、2から、1に、設定し直しました。

@seii-saintway

This comment has been minimized.

Copy link
Contributor Author

@seii-saintway seii-saintway commented Dec 2, 2019

@oss-gate-issue-cleaner

This comment has been minimized.

Copy link

@oss-gate-issue-cleaner oss-gate-issue-cleaner bot commented Dec 3, 2019

おつかれさまでした!

後日引き続きワークショップがある場合

後日のワークショップで再開する際に、issueタイトルの日付を更新してreopenしてください!

ワークショップ終了後に続きがしたい場合

issueはクローズしますが、その後も作業を続けていただいて大丈夫です!

是非このissueを活用なさってください。

今後のイベントの案内

現在以下のイベントの開催を予定しています。

参加をお待ちしてます!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.