Negsignal.SIGABRT when trying to create matplotlib plots in DAG

[avriiil]

Apache Airflow version

2.2.2 (latest released)

What happened

My airflow tasks which creates matplotlib figures and saves them locally fails with the following error in the logs:
INFO - Task exited with return code Negsignal.SIGABRT

full traceback:

*** Reading local file: /Users/rpelgrim/airflow/logs/airflow_on_coiled_testing/visualize/2021-12-08T12:31:14.863577+00:00/1.log
[2021-12-08, 13:35:23 UTC] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: airflow_on_coiled_testing.visualize manual__2021-12-08T12:31:14.863577+00:00 [queued]>
[2021-12-08, 13:35:23 UTC] {taskinstance.py:1035} INFO - Dependencies all met for <TaskInstance: airflow_on_coiled_testing.visualize manual__2021-12-08T12:31:14.863577+00:00 [queued]>
[2021-12-08, 13:35:23 UTC] {taskinstance.py:1241} INFO - 
--------------------------------------------------------------------------------
[2021-12-08, 13:35:23 UTC] {taskinstance.py:1242} INFO - Starting attempt 1 of 4
[2021-12-08, 13:35:23 UTC] {taskinstance.py:1243} INFO - 
--------------------------------------------------------------------------------
[2021-12-08, 13:35:23 UTC] {taskinstance.py:1262} INFO - Executing <Task(_PythonDecoratedOperator): visualize> on 2021-12-08 12:31:14.863577+00:00
[2021-12-08, 13:35:23 UTC] {standard_task_runner.py:52} INFO - Started process 33014 to run task
[2021-12-08, 13:35:23 UTC] {standard_task_runner.py:76} INFO - Running: ['airflow', 'tasks', 'run', 'airflow_on_coiled_testing', 'visualize', 'manual__2021-12-08T12:31:14.863577+00:00', '--job-id', '77', '--raw', '--subdir', 'DAGS_FOLDER/airflow-coiled-testing.py', '--cfg-path', '/var/folders/ky/bqjn_gxn1xv0cn_8q5xvp3q40000gn/T/tmpzzle0cas', '--error-file', '/var/folders/ky/bqjn_gxn1xv0cn_8q5xvp3q40000gn/T/tmp9kwl1ww2']
[2021-12-08, 13:35:23 UTC] {standard_task_runner.py:77} INFO - Job 77: Subtask visualize
[2021-12-08, 13:35:23 UTC] {logging_mixin.py:109} INFO - Running <TaskInstance: airflow_on_coiled_testing.visualize manual__2021-12-08T12:31:14.863577+00:00 [running]> on host 1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.ip6.arpa
[2021-12-08, 13:35:23 UTC] {taskinstance.py:1427} INFO - Exporting the following env vars:
AIRFLOW_CTX_DAG_EMAIL=richard@coiled.io
AIRFLOW_CTX_DAG_OWNER=rrpelgrim
AIRFLOW_CTX_DAG_ID=airflow_on_coiled_testing
AIRFLOW_CTX_TASK_ID=visualize
AIRFLOW_CTX_EXECUTION_DATE=2021-12-08T12:31:14.863577+00:00
AIRFLOW_CTX_DAG_RUN_ID=manual__2021-12-08T12:31:14.863577+00:00
[2021-12-08, 13:35:23 UTC] {local_task_job.py:154} INFO - Task exited with return code Negsignal.SIGABRT
[2021-12-08, 13:35:23 UTC] {taskinstance.py:1270} INFO - Marking task as UP_FOR_RETRY. dag_id=airflow_on_coiled_testing, task_id=visualize, execution_date=20211208T123114, start_date=20211208T123523, end_date=20211208T123523
[2021-12-08, 13:35:23 UTC] {local_task_job.py:264} INFO - 0 downstream tasks scheduled from follow-on schedule check

What you expected to happen

For the task to complete (since it runs fine locally in a jupyter notebook running on the same software env) and for the figure to be plotted and stored locally.

How to reproduce

Reproduce with this DAG:

from datetime import datetime, timedelta
import matplotlib.pyplot as plt
import numpy as np

from airflow.decorators import dag, task

# set default arguments to all tasks
default_args = {
    'owner': 'rrpelgrim',
    'depends_on_past': False,
    'email_on_failure': False,
    'email_on_retry': False,
    'retries': 3,
    'retry_delay': timedelta(minutes=5),
}

@dag(
    default_args=default_args,
    schedule_interval=None, 
    start_date=datetime(2021, 1, 1),
    catchup=False,
    )
def dag_name():
    
    @task()
    def visualize():
        """
        Create visualisation plots and save as PNGs.
        """
        # Create boxplot 
        data = np.arange(1,20)
        fig, ax = plt.subplots(figsize=(10,10)) 
        plt.title('Title', fontsize=15)
        plt.boxplot(x=data)       
        plt.savefig('boxplot_test.png')
        plt.close(fig)

        print('Visualisations have been created.')

    # Call task functions in order
    viz = visualize()

# Call taskflow
demo = airflow_on_coiled_testing()

Operating System

macOS 11.3.1

Versions of Apache Airflow Providers

No response

Deployment

Other

Deployment details

working locally with airflow standalone

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[potiuk]

There is not much information but from your description it looks like this (or similar): https://stackoverflow.com/questions/60574601/how-can-i-fix-a-sigabrt-error-on-my-macos-hight-sierra

Simply Airflow tasks run as non-interactive process and they have no acces to your graphical card (and your UI). The opencv-headless package allows you to build the visualisation using only CPU.

You could also use your GPU likely but that requires some more complex setting and running an in-memory X Server with GPU acceleration (but you 'd need to dig much deeper to get it working).

[avriiil]

Thanks @potiuk. Installing the opencv-python-headless package did not solve it for me, unfortunately.
I'm hesitant to dig into running it on my GPU. Do you jappen to know of another simpler workaround?

[potiuk]

You should really check what's the error message there. You can try to run your stuff for example from a cron job to avoid the "interactive" part.

[potiuk]

And I am moving that to discussion - this is not really an Airlfow bug.