Plan a strategy to gracefully handle dataframes so users don't trash Airflow metadata database

[tatiana]

Context

At the moment the dataframe decorator adds the entire content of a dataframe to Airflow XCom. Depending on how the user set XCom, this can potentially trash Airflow's metadata database (if the user is using the default XCom backend).

There are a few approaches we could use to deal with this:

• Give warnings to users if they use the dataframe decorator with the default backend
• Not allow users to use the dataframe decorator with the default backend if the dataframe is larger than a specific size
• Others

The goal with this ticket is propose a solution for the problem.

Acceptance criteria

• Log a ticket with the agreed approach

[dimberman]

@tatiana I think one solution here would be to force users to explicitly set an environment variable of "yes I know that I'm putting dataframes in the airlfow metadata and yes I know that's dangerous". Something like "ASTRO__DATAFRAME__ALLOW_UNSAFE_STORAGE" cc: @kaxil