rendered_task_instance_fields insert failure

[electrical]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

Sometimes with dag runs we see a Postgres error with inserting data into the rendered_task_instance_fields table.
This does not happen consistently but feels like it happens when there is a lot of data to be processed.

See logs: https://gist.github.com/electrical/6b5d337dd703b8cf0d06d79b06bd51ee

What you think should happen instead

Expect it not to fail

How to reproduce

Standard Airflow setup
Either manual or scheduled run

Operating System

Bitnami Debian docker image

Versions of Apache Airflow Providers

No response

Deployment

Other 3rd-party Helm chart

Deployment details

Apache airflow 2.5.3
Kubernetes 1.24.13
Using Bitnami Helm chart

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[hussein-awala]

There might be an issue in your database that is preventing it from processing the load. It seems that the connection is being closed for some reason. Possible causes for this issue could include:

• The database server is not running properly and cannot accept connections.
• There is a network connectivity problem between the Airflow server and the PostgreSQL server.
• There is a performance issue in the database, and the connection timeout value in the Airflow configuration is shorter than the processing time. You can try increasing the connection timeout value by providing {'connect_timeout': <new timeout>} to this conf).

Can you check if there is more information in the database queries/connections logs and dashboard?

I convert it to a discussion, and we can reopen it if needed.

[electrical]

When we see this error we see the following in the postgres logs:

2023-05-31T09:11:03.718529605Z 2023-05-31 09:11:03.718 GMT [1] DETAIL:  Failed process was running: INSERT INTO rendered_task_instance_fields (dag_id, task_id, run_id, map_index
, rendered_fields, k8s_pod_yaml) VALUES ('pricing_imti_rates_to_clickhouse', 'load_records', 'manual__2023-05-31T09:09:44+00:00',  -1, '{"templates_dict": null, "op_args": ["[{\"organization_id\": 59676, \"buy_currency_code\": \"GBP\", \"sell_currency_code\": \"JPY\", \"country_from_iso2_code\": \"GB\", \"is_individual\": true, \"is_sending\": true, \"payment_method\": \"Bank Account\", \"payout_method\": \"Bank Account\", \"account_type\": \"Standard Consumer Account\", \"data_validation_record_id\": 22400, \"account_name\": \"Cashminder\", \"countries_to\": [], \"amounts_and_rates\": [{\"amount_from\": 100, \"fx_rate\": 137.9576, \"fx_margin\": 4.066867863649637, \"interbank_rate\": 143.806, \"scraping_time_utc\": {\"$date\": 1612179660000}, \"amount_to\": 0}, {\"amount_from\": 1000, \"fx_rate\": 137.9576, \"fx_margin\": 4.066867863649637, \"interbank_rate\": 143.806, \"scraping_time_utc\": {\"$date\": 1612179660000}, \"amount_to\
2023-05-31T09:11:03.718546725Z 2023-05-31 09:11:03.718 GMT [1] LOG:  terminating any other active server processes
2023-05-31 09:11:03.790 GMT [1] LOG:  all server processes terminated; reinitializing
2023-05-31T09:11:03.912169561Z 2023-05-31 09:11:03.912 GMT [2286252] LOG:  database system was interrupted; last known up at 2023-05-31 09:08:57 GMT