New job fail immediately after start, no logs

[rchangj]

Apache Airflow version:1.10.12

Apache Airflow Provider versions (please include all providers that are relevant to your bug):
No other providers

Kubernetes version (if you are using kubernetes) (use kubectl version):
No using k8s

Environment:
AWS Linux

• Cloud provider or hardware configuration:AWS
• OS (e.g. from /etc/os-release):
  -NAME="Amazon Linux"
  VERSION="2"
  ID="amzn"
  ID_LIKE="centos rhel fedora"
• Kernel (e.g. uname -a): 4.14.186-146.268.amzn2.x86_64 Improving the search functionality in the graph view #1 SMP Tue Jul 14 18:16:52 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: yum, pip3
• Others:

What happened:
I have been running airflow 1.10.12 for several months without any issues. Recently I updated airflow to use SSL cert. After that I reboot the airflow server and then restarted the airflow scheduler and Webserver as following:
airflow Webserver &
airflow scheduler &

The jobs run normally at beginning. However, within one day, all the new jobs will immediately fail. From UI, I saw the task status as failed. However there is no log under the task when I checked the log from UI.

I also checked the /logs/scheduler directory, except a warning of "psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "variable_key_key", which has existed for long time, I didn't see any other warning or error.

Now this problem happened every day, i.e., if I restart airflow Webserver and scheduler, it works well. But within one day, all the new jobs will immediately fail with no log.

Do you have any insight what can go wrong? Thanks.

What you expected to happen:

How to reproduce it:
Not exactly sure how to reproduce. I just reboot airflow server.

Anything else we need to know:

How often does this problem occur? Once? Every time etc?
I noticed the problem happens within 1 day.

Any relevant logs to include? Put them here in side a detail tag:

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

It's really, really hard to say without more logs, You might want to try to find more logs. Also It would be great if you explain what kind of SSL configuration you've made (why?) and possibly if you could revert that change - even in a test system) and see if that fixes the problem. I do not expect this, maybe you also upgraded some libraries etc. along the way.

Just one note - it's not very likely you will get a lot of help with 1.10.12. Airflow 1.10 reached End-Of-Life on June 17th 2021 and It will not receive any more fixes - not even the critical ones, so you should upgrade to Airflow 2 as soon as you can. Maybe this is the right time.

[rchangj]

The SSL certs are just self signed certs created as following.
openssl genrsa -out private.pem 2048
openssl req -new -x509 -key private.pem -out cacert.pem -days 1095
The airflow.cfg changes are as followings:
web_server_ssl_cert = path/to/cacert.pem
web_server_ssl_key = path/to/private.pem

There are ABSOLUTELY no other changes nor any other lib installation.

[rchangj]

After I use following command to start airflow, the issue goes away. It seems there is a corner issue due to permission:
sudo /sbin/runuser -l -c "airflow scheduler"

Meantime, I have airflow 2.1.2 installed and running the scheduler, finger crossed.

[david30907d]

bumped into the same issue, DummyOperator failed immediately with any log

[eladkal]

bumped into the same issue, DummyOperator failed immediately with any log

There isn't enough information to understand what is the issue.
Please add reproduce steps.
I would suggest to file a new issue as this one is on Airflow 1.10.12 which is EOL and author reported the issue has been resolved.

[david30907d]

bumped into the same issue, DummyOperator failed immediately with any log

There isn't enough information to understand what is the issue.
Please add reproduce steps.
I would suggest to file a new issue as this one is on Airflow 1.10.12 which is EOL and author reported the issue has been resolved.

Hi @eladkal any suggestions to get the log? I'm using 2.1.3 and have no idea how to provide more details

[github-actions]

This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.

[rchangj]

This issue shows on version 2.1.2 as well. There is no logs of any error shows up. As I mentioned earlier, following command "sudo /sbin/runuser -l -c "airflow scheduler" can be a work around. It would be good to have a fix for this issue.

[github-actions]

This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.