Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-6505] Let emoji encoded properly for json.dumps() -- BaseSQ… #1

Closed
wants to merge 1,276 commits into from

Conversation

damon09273
Copy link
Owner

@damon09273 damon09273 commented Jan 8, 2020

Make sure you have checked all steps below.

Jira

Description

Emoji can't be encoded properly when json.dumps() and 'UTF-8', problem fixed by adding parameter ensure_ascii=False.
In line BaseSQLToGoogleCloudStorageOperato.

For example

the emoji 🍻 encoded differentially when in use or not use ensure_ascii=False.

In Use (correct UTF-8 encode): "\xf0\x9f\x8d\xbb"

Not Use:"\\ud83c\\udf7b"

Ref: https://stackoverflow.com/questions/51183947/python-json-dumps-doesnt-encode-emojis-properly

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain docstrings that explain what it does
    • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

  • Passes flake8

Make sure you have checked all steps below.

Jira

  • My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
    • https://issues.apache.org/jira/browse/AIRFLOW-XXX
    • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.
    • In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal (AIP).
    • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Description

  • Here are some details about my PR, including screenshots of any UI changes:

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

  • My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain docstrings that explain what it does
    • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

  • Passes flake8

michalslowikowski00 and others added 30 commits January 8, 2020 12:08
PR contains changes regarding AIP-21 (renaming GCP operators and hooks):

* renamed GCP modules
* adde deprecation warnings to the contrib modules
* fixed tests
* updated UPDATING.md
The containers were not removed and you have to remove them
with `dockery system prune`. The --rm flag is added.
…lls (apache#7089)

no need for 2 sleep, _processor_poll_interval will already sleep
…7112)

PR contains changes regarding AIP-21 (renaming GCP operators and hooks):
* renamed GCP modules
* fixed tests
Add dag_ids GET parameter to /blocked end point to allow querying of just
the dags present on the current page.
… be skippable by config option (apache#7096)" (apache#7129)

This reverts commit 77b1bdc.

Reverts apache#7096 to do in a slightly different way (without a new config option), and reverting this so that the new change is easier to backport to 1.10 releases.
potiuk and others added 24 commits February 6, 2020 13:52
* [AIRFLOW-6733] Extend, rather than replace, the base_template

This commit just moves the existing template to airflow/master.html
(without further changes)

* [AIRFLOW-6733] Only change blocks we have customized

This makes is easier to see which parts of the template we have changed.
…pache#7367)

Flask/FAB gives us a `base_template` variable that we should use instead
of hard-coding a specific template to extend.
…apache#7371)

The logs are now sent in the bash scripts when all tests are complete
If the number of dags was large and/or the length of the DAG ids were too large this would exceed the maximum possible query string limit.

To work around that we have made these endpoints always make POST requests
Unknown parameter in input: "Workgroup", must be one of: QueryString, ClientRequestToken, QueryExecutionContext, ResultConfiguration, WorkGroup
…che#7382)

This happens when you have shared clone of the repository
…ache#7318)

* [AIRFLOW-XXXX] Add explicit info about JIRAs for code-related PRs

* fixup! [AIRFLOW-XXXX] Add explicit info about JIRAs for code-related PRs

Co-Authored-By: Jarek Potiuk <jarek@potiuk.com>

Co-authored-by: Jarek Potiuk <jarek@potiuk.com>
…pache#7393)

* [AIRFLOW-6766] Fix "cannot import ensure_text" error for pre-commit

As of today Travis bundles six version 1.11.0 with their python
3.6 image and it misses ensure_text method. Bumping to 1.14+
solves the problem.
* Change default wg name

* Change workgroup in tests
damon09273 pushed a commit that referenced this pull request Feb 11, 2020
…n using job_flow_name and no cluster is found (apache#6898)

* [AIRFLOW-6432] fixes in EmrAddStepsOperator

fix EmrAddStepsOperator broken ref & faulty test

* changes after CR #1

* Add exception and test case

* Update airflow/contrib/hooks/emr_hook.py

Co-Authored-By: Tomek Urbaszek <turbaszek@gmail.com>

* Update airflow/contrib/hooks/emr_hook.py

Co-Authored-By: Tomek Urbaszek <turbaszek@gmail.com>

* Update airflow/contrib/operators/emr_add_steps_operator.py

Co-Authored-By: Tomek Urbaszek <turbaszek@gmail.com>

* Update airflow/contrib/hooks/emr_hook.py

Co-Authored-By: Tomek Urbaszek <turbaszek@gmail.com>

* Update tests/contrib/operators/test_emr_add_steps_operator.py

Co-Authored-By: Tomek Urbaszek <turbaszek@gmail.com>

* changes after CR apache#2

Co-authored-by: Tomek Urbaszek <turbaszek@gmail.com>
@damon09273 damon09273 closed this Feb 11, 2020
@damon09273 damon09273 reopened this Feb 11, 2020
@damon09273 damon09273 closed this Feb 18, 2020
@damon09273 damon09273 deleted the fix_emoji_encoding_sql_to_gcs branch February 18, 2020 05:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet