Skip to content

Add airflow error-guide command#46932

Draft
omkar-foss wants to merge 1 commit intoapache:mainfrom
omkar-foss:feat-airflow-guide
Draft

Add airflow error-guide command#46932
omkar-foss wants to merge 1 commit intoapache:mainfrom
omkar-foss:feat-airflow-guide

Conversation

@omkar-foss
Copy link
Collaborator

closes: #43171
related: #44616

This PR adds the airflow error-guide command as per the plan in this comment.

@omkar-foss
Copy link
Collaborator Author

Guide for specific error code:

$ airflow error-guide --error-code AERR100

The error you're facing would be with this message: "Invalid DAG structure".

As per our observations, a possible cause could be as follows:
Triggered when a DAG's dependencies or attributes are incorrectly defined.

To resolve this, as first step, you can try the following:
Review the DAG structure for errors. Ensure that all dependencies are correctly defined and that no circular dependencies 
exist.

If this doesn't resolve your problem, you can check out the docs for more info:
https://airflow.apache.org/docs/apache-airflow/stable/concepts/dags.html

You may also ask your questions on the Airflow Slack #user-troubleshooting channel:
https://apache-airflow.slack.com/messages/user-troubleshooting

Happy Debugging! 🐞

Listing all exceptions covered in guide:

airflow error-guide --list-exceptions   
Exception Type           
=========================
AirflowApiException      
AirflowCliException      
AirflowConfigException   
AirflowDagCycleException 
AirflowDagImportException
AirflowDagNotFound       
AirflowDagPausedException
AirflowDatabaseException 
AirflowException         
AirflowExecutorException 
AirflowParseException    
AirflowSchedulerException
AirflowTaskException     
AirflowTaskTimeout       
AirflowTemplateException 
AirflowTriggerException  
AirflowWebServerException
AirflowWorkerException   
AirflowXComException     
AttributeError           
FileNotFoundError        
ImportError              
KeyError                 
ModuleNotFoundError      
PermissionError          
UnpicklingError          
ValueError               

Dump entire guide as yaml:

airflow error-guide --list-guide
AERR001:
  description: Happens when dynamically mapped tasks exceed the maximum number of
    tasks allowed.
  documentation: https://airflow.apache.org/docs/apache-airflow/stable/concepts/dynamic-task-mapping.html
  error_message: Dynamic task mapping exceeded limit
  exception_type: AirflowException
  first_steps: Check the task count limit in the configuration. Consider increasing
    the task limit or optimizing task mapping logic.
AERR002:
  description: Happens when the scheduler or webserver cannot locate a task instance
    in the database.
  documentation: https://airflow.apache.org/docs/apache-airflow/stable/administration/task-instances.html
  error_message: Task instance not found
  exception_type: AirflowException
  first_steps: Verify that the database connection is stable. Check if the task instances
    exist in the metadata database and consider re-running the DAG.
...truncated

@omkar-foss omkar-foss self-assigned this Feb 20, 2025
@omkar-foss omkar-foss force-pushed the feat-airflow-guide branch 3 times, most recently from 2ef45ed to 44ff5f8 Compare February 26, 2025 04:58
@potiuk
Copy link
Member

potiuk commented Mar 30, 2025

There is some chance we will look at it as one of the first things after all 'critical' airflow 3 things are done. So do not loose your faith :)

@omkar-foss
Copy link
Collaborator Author

omkar-foss commented Apr 1, 2025

There is some chance we will look at it as one of the first things after all 'critical' airflow 3 things are done. So do not loose your faith :)

Sure got it, I'll try my best to complete this PR till then. Thanks for the assurance, appreciate it :)

@omkar-foss omkar-foss force-pushed the feat-airflow-guide branch 2 times, most recently from efc4b9d to 2c37ce3 Compare April 1, 2025 10:11
@omkar-foss omkar-foss force-pushed the feat-airflow-guide branch from 2c37ce3 to 6cb35b3 Compare May 14, 2025 05:53
@github-actions
Copy link

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Jun 29, 2025
@github-actions github-actions bot closed this Jul 27, 2025
@potiuk potiuk reopened this Jul 27, 2025
@potiuk
Copy link
Member

potiuk commented Jul 27, 2025

@omkar-foss -> should we revive it ? I thought it was a cool idea and we should make it happen eventually

@github-actions github-actions bot removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Jul 28, 2025
@uranusjr
Copy link
Member

Should this be in airflowctl instead? It does not need anything specific from core.

@potiuk
Copy link
Member

potiuk commented Jul 30, 2025

Should this be in airflowctl instead? It does not need anything specific from core.

The idea was that (eventually) those constants will be used when any exception is raised - so they would eventually have to be imported from somewhere. For me that is likely a better candidate for "shared/errors" library rather than "airfow-ctl" code.

@omkar-foss
Copy link
Collaborator Author

omkar-foss commented Aug 22, 2025

@omkar-foss -> should we revive it ? I thought it was a cool idea and we should make it happen eventually

Sure, let's revive this. We could continue with our initial thought of having AirflowEnumeratedException class with compulsory error_id getting validated against a yaml file (source of truth) containing error ids info as described in this comment.

airflowctl 'error-guide' command in this PR could be a way for users to access error id info easily (without looking at yaml file), but that'd be after we've the main error id framework in place.

@potiuk wdyt?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make Airflow error messages more specific, clear and actionable

3 participants