Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GenericTransfer - Warning: "Placeholder defined in Connection..." #39678

Closed
1 of 2 tasks
romanzdk opened this issue May 17, 2024 · 4 comments · Fixed by #39690
Closed
1 of 2 tasks

GenericTransfer - Warning: "Placeholder defined in Connection..." #39678

romanzdk opened this issue May 17, 2024 · 4 comments · Fixed by #39690

Comments

@romanzdk
Copy link

Apache Airflow version

2.9.1

If "Other Airflow 2 version" selected, which one?

No response

What happened?

We use GenericTransfer task to copy data from one DB to another. Task is successful but we get a warning in logs:
[2024-05-17, 10:08:43 UTC] {sql.py:187} WARNING - Placeholder defined in Connection 'None' is not listed in 'DEFAULT_SQL_PLACEHOLDERS' and got ignored. Falling back to the default placeholder '%s'.

I checked the Connections in the Admin and everything looks alright.

Screenshot 2024-05-17 at 12 22 33

What you think should happen instead?

No warning log

How to reproduce

create and run GenericTransfer task:

copy_task = GenericTransfer(
	task_id = 'copy_task',
	dag = dag,
	source_conn_id = 'source-db',
	destination_conn_id = 'bi-database',
	sql = '''
		SELECT
			"date"
		FROM "TableName"
	''',
	destination_table = '"TableName"',
	preoperator = '''
		DELETE FROM "TableName"
		WHERE
			1 = 1
	'''
)

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.20.0
apache-airflow-providers-celery==3.6.2
apache-airflow-providers-cncf-kubernetes==8.1.1
apache-airflow-providers-common-io==1.3.1
apache-airflow-providers-common-sql==1.12.0
apache-airflow-providers-fab==1.0.4
apache-airflow-providers-ftp==3.8.0
apache-airflow-providers-http==4.10.1
apache-airflow-providers-imap==3.5.0
apache-airflow-providers-postgres==5.10.2
apache-airflow-providers-smtp==1.6.1
apache-airflow-providers-sqlite==3.7.1

Deployment

Other Docker-based deployment

Deployment details

Airflow 2.9.1, on-premise Kubernetes cluster 1.26.5, python 3.10.12, Celery executor, Postgres 13.13

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

@romanzdk romanzdk added area:core kind:bug This is a clearly a bug needs-triage label for new issues that we didn't triage yet labels May 17, 2024
@Taragolis
Copy link
Contributor

@Taragolis Taragolis added area:providers provider:common-sql good first issue and removed area:core needs-triage label for new issues that we didn't triage yet labels May 17, 2024
@Taragolis
Copy link
Contributor

I see two problem here:

  1. An error in a warning message, instead of connection id use placeholder
  2. The warning raised in case if placeholder not defined in connection

@dabla
Copy link
Contributor

dabla commented May 17, 2024

I'm on it, will write a test to reproduce but I suppose a None check is missing hence why the warning message.

@dabla
Copy link
Contributor

dabla commented May 17, 2024

I've created a PR for this which fixes the issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants