-
Notifications
You must be signed in to change notification settings - Fork 16.6k
Description
Apache Airflow version: Going from a working 1.10.12 -> .14
Environment:
- Cloud provider or hardware configuration: GCP
- OS (e.g. from /etc/os-release): Debian
- Kernel (e.g.
uname -a): 4.19.0-10-cloud-amd64 Improving the search functionality in the graph view #1 SMP Debian 4.19.132-1 (2020-07-24) x86_64 GNU/Linu
What happened:
I have a mix of Terraform/Ansible to handle GCP and the installation on VM. Everything was working good with 1.10.12 and I attemped to update to 1.10.14. The webserver doesn't want to start because of this error :
airflow.exceptions.AirflowConfigException: section/key [google/prompt] not found in config
I can find here in the documentation that part :
https://airflow.apache.org/docs/apache-airflow/1.10.14/security.html#google-authentication
I never had to setup [google/prompt] and I can see it appear in the .13 documentation
It's from this PR : #11890
The documentation is not clear enough for me. So I put what seems to me the less scary : "none".
But now I have an error when I try to init the db :
fatal: [sea-cloud-airflow-applications]: FAILED! => {"msg": "the field 'become' has an invalid value ({{airflow_user}}), and could not be converted to an bool.The error was: The value 'airflow' is not a valid boolean. Valid booleans include: 0, 1, 'no', 'on', 'f', 't', 'off', '0', '1', 'true', 'false', 'n', 'y', 'yes'"}
When I see the journalctl I got this :
____________ _____________
____ |__( )_________ __/__ /________ __
____ /| |_ /__ ___/_ /_ __ /_ __ \_ | /| / /
___ ___ | / _ / _ __/ _ / / /_/ /_ |/ |/ /
_/_/ |_/_/ /_/ /_/ /_/ \____/____/|__/
{__init__.py:50} INFO - Using executor SequentialExecutor
{dagbag.py:417} INFO - Filling up the DagBag from /home/airflow/airflow/dags
{dagbag.py:259} ERROR - Failed to import: /usr/local/lib/python3.7/dist-packages/airflow/example_dags/example_subdag_operator.py
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: slot_pool
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/airflow/models/dagbag.py", line 256, in process_file
m = imp.load_source(mod_name, filepath)
File "/usr/lib/python3.7/imp.py", line 171, in load_source
module = _load(spec)
File "<frozen importlib._bootstrap>", line 696, in _load
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/usr/local/lib/python3.7/dist-packages/airflow/example_dags/example_subdag_operator.py", line 51, in <module>
dag=dag,
File "/usr/local/lib/python3.7/dist-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/airflow/utils/decorators.py", line 98, in wrapper
result = func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/airflow/operators/subdag_operator.py", line 77, in __init__
.filter(Pool.pool == self.pool)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/orm/query.py", line 3429, in first
ret = list(self[0:1])
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/orm/query.py", line 3203, in __getitem__
return list(res)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
distilled_params,
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: slot_pool
[SQL: SELECT slot_pool.id AS slot_pool_id, slot_pool.pool AS slot_pool_pool, slot_pool.slots AS slot_pool_slots, slot_pool.description AS slot_pool_description
FROM slot_pool
WHERE slot_pool.slots = ? AND slot_pool.pool = ?
LIMIT ? OFFSET ?]
[parameters: (1, 'default_pool', 1, 0)]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: connection
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/airflow", line 37, in <module>
args.func(args)
File "/usr/local/lib/python3.7/dist-packages/airflow/utils/cli.py", line 81, in wrapper
return f(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/airflow/bin/cli.py", line 1179, in webserver
app = cached_app_rbac(None) if settings.RBAC else cached_app(None)
File "/usr/local/lib/python3.7/dist-packages/airflow/www/app.py", line 244, in cached_app
app = create_app(config, testing)
File "/usr/local/lib/python3.7/dist-packages/airflow/www/app.py", line 99, in create_app
from airflow.www import views
File "/usr/local/lib/python3.7/dist-packages/airflow/www/views.py", line 2504, in <module>
class ChartModelView(wwwutils.DataProfilingMixin, AirflowModelView):
File "/usr/local/lib/python3.7/dist-packages/airflow/www/views.py", line 2595, in ChartModelView
'conn_id': _connection_ids()
File "/usr/local/lib/python3.7/dist-packages/airflow/utils/db.py", line 74, in wrapper
return func(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/airflow/www/views.py", line 2501, in _connection_ids
.group_by(Connection.conn_id))]
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/orm/query.py", line 3535, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/orm/query.py", line 3560, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1011, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1130, in _execute_clauseelement
distilled_params,
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1317, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1511, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/util/compat.py", line 182, in raise_
raise exception
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1277, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: connection
[SQL: SELECT connection.conn_id AS connection_conn_id
FROM connection GROUP BY connection.conn_id]
(Background on this error at: http://sqlalche.me/e/13/e3q8)
airflow-webserver.service: Main process exited, code=exited, status=1/FAILURE
airflow-webserver.service: Failed with result 'exit-code'.
airflow-webserver.service: Service RestartSec=5s expired, scheduling restart.
airflow-webserver.service: Scheduled restart job, restart counter is at 358.
Stopped Airflow webserver daemon.