Add reason for enabling XCOM pickling #747
Conversation
sunank200
requested review from
dimberman,
tatiana,
utkarsharma2,
pankajastro,
pankajkoti and
feluelle
as code owners
Codecov Report
@@ Coverage Diff @@
## main #747 +/- ##
=======================================
Coverage 93.26% 93.26%
=======================================
Files 43 43
Lines 1855 1855
Branches 232 232
=======================================
Hits 1730 1730
Misses 97 97
Partials 28 28 Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here. |
Co-authored-by: Daniel Imberman <daniel.imberman@gmail.com>
for more information, see https://pre-commit.ci
sunank200
force-pushed
the
document-enabling-XCom-pickling-reason
branch
from
0eec9e7
to
f548961
Compare
|
@dimberman please re-review |
README.md
Outdated
|
|
||
| Currently, custom XCom backends are limited to data types that are json serializable. Since Dataframes are not json serializable, we need to enable XCom pickling to store dataframes. | ||
|
|
||
| The data format used by pickle is Python-specific. This has the advantage that there are no restrictions imposed by external standards such as JSON or XDR (which can’t represent pointer sharing); however it means that non-Python programs may not be able to reconstruct pickled Python objects. |
There was a problem hiding this comment.
It would add in context to user for why we need this.
There was a problem hiding this comment.
however it means that non-Python programs may not be able to reconstruct pickled Python objects.
I mean this line, we don't need that one ^^. But I will let you decide
There was a problem hiding this comment.
Maybe if someone wants to opt-out and use airflow rest api?
There was a problem hiding this comment.
I did not understand @feluelle. Could you elaborate on this, please?
There was a problem hiding this comment.
If you read data from xcom via the airflow rest api you would not be able to reconstruct it or? Maybe only with a python rest api client - but not the go client for example or? (that is what the sentence is saying, right? So maybe it is useful to leave it there.)
|
Docs build is still failing - https://github.com/astronomer/astro-sdk/runs/8207855916?check_suite_focus=true |
|
Can you rebase your branch on |
|
|
@kaxil please re-review this. I think its good to merge. |
Description
What is the current behavior?
Missing reason in docs and readme for enabling XCOM pickling
closes: #739
What is the new behavior?
Does this introduce a breaking change?
No
Checklist