Mysql change xcom value col type for MySQL backend

[pankajastro]

Xcom value we can store Airflow metadata is a bit smaller
when we use MySQL as a database backend since by default,
the Sqlalchemy map LargeBinary to MySQL blob type
which can store up to 65,535 bytes. In this PR,
I'm proposing to change the value column type to
longblob for xcom table if a user using MySQL database backend
https://dev.mysql.com/doc/refman/8.0/en/string-type-syntax.html

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

Will that cause a long migration process possibly ? or will such migration be rather fast ? I suspect if there is something to rewrite the way how the blobs are stored, it might take an awfully long time to run such a migration.

I think, if that's the case then we will likely have to at least warn the users in significant note that it might happen and that they should likely run db clean on xcom table for some old data?

[pankajastro]

Will that cause a long migration process possibly ?

Yes, that will be the case if they have lots of large data in Xcom, although I don't have a specific number at the moment.

I think, if that's the case then we will likely have to at least warn the users in significant note that it might happen and that they should likely run db clean on xcom table for some old data?

This makes sense. I'll add a warning.
If the blob size > 65,535 bytes then to downgrade they will have to clean at least large data.

[pankajastro]

@potiuk I have tried to add a note docs/apache-airflow/howto/set-up-database.rst but not sure if this right place. any suggestions?

[potiuk]

Should be a significant newsfragment too.

[jedcunningham]

We should definitely test whether this will rewrite all of xcom - it might not.

If it does, and it results in a slow migration, I really wonder if we should not do this and instruct users to set up an object storage xcom backend instead. Thoughts?

[Taragolis]

I think, if that's the case then we will likely have to at least warn the users in significant note that it might happen and that they should likely run db clean on xcom table for some old data?

A bit Friday jokes: If take in account last survey - number of MySQL users about 11-14 % (11% MySQL 8, and 3.4% MySQL 5) for avoid long migration it could be an option to start from scratch with Postgres 🤣 .

[potiuk]

I think, if that's the case then we will likely have to at least warn the users in significant note that it might happen and that they should likely run db clean on xcom table for some old data?

A bit Friday jokes: If take in account last survey - number of MySQL users about 11-14 % (11% MySQL 8, and 3.4% MySQL 5) for avoid long migration it could be an option to start from scratch with Postgres 🤣 .

Yes. Let's also add "if mysql: sleep(0.2)" in various places in Airlfow. Enough to be impacting performance, but not enough to say "MYSQL does not work". Then we will be able to tell "But Postgres is WAY faster".

[kaxil]

If it does, and it results in a slow migration, I really wonder if we should not do this and instruct users to set up an object storage xcom backend instead. Thoughts?

The current size is low per row - 64kb for MySQL, so increasing to 64 MB should be supported without Custom XCom backend