Add valid spark on k8s airflow connection urls to tests #31376

amoghrajesh · 2023-05-18T08:34:29Z

Airflow connections parses valid urls fine. The issue described in #21638 provided a wrong url as input to the connections with double :// leading to parsing issues. This PR simply adds a few valid spark on k8s connection urls to the existing test suite.

Testing by adding various different unit test cases to cover the breadth

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

uranusjr · 2023-05-18T08:55:19Z

airflow/models/connection.py

+        uri = [e for e in uri]
+        # replace back :__ with ://
+        uri[1] = uri[1].replace(":__", "://")
+        uri = SplitResult(uri[0], uri[1], uri[2], uri[3], uri[4])


Why do you need to do this, uri is already a SplitResult instance…?

Thanks for your comment. No, one line 200 we are doing a list comprehension. This is leading to the type being as list.

Verified by adding a few log statements:

def split_uri_to_parts(uri): c = uri.count("://") if c < 2: return urlsplit(uri), False idx = uri.rfind("://") # replace :// with :__ uri_list = list(uri) uri_list[idx : idx + 3] = ":__" uri = "".join(uri_list) uri = urlsplit(uri) uri = [e for e in uri] # replace back :__ with :// uri[1] = uri[1].replace(":__", "://") print(type(uri)) uri = SplitResult(uri[0], uri[1], uri[2], uri[3], uri[4]) print(type(uri)) return uri, True

Output is:

<class 'list'> <class 'urllib.parse.SplitResult'> (SplitResult(scheme='spark', netloc='k8s://100.68.0.1:443', path='', query='deploy-mode=cluster', fragment=''), True)

And why do you need to do that list comprehention in the first place? SplitResult supports index access out of the box.

Hello, I am trying to re-assign a variable to SplitResult list. That list does not support reassignment. Hence to achieve something like that, I had to convert it to a list, re-assign the elements and then convert it back to the SplitResult list. @uranusjr

/Users/adesai/.pyenv/versions/airflow-env/bin/python /Users/adesai/Documents/OpenSource/Airflow/airflow/airflow/models/test.py Traceback (most recent call last): File "/Users/adesai/Documents/OpenSource/Airflow/airflow/airflow/models/test.py", line 24, in <module> parts = split_uri_to_parts("spark://k8s://100.68.0.1:443?deploy-mode=cluster") File "/Users/adesai/Documents/OpenSource/Airflow/airflow/airflow/models/test.py", line 15, in split_uri_to_parts uri[1] = uri[1].replace(":__", "://") TypeError: 'SplitResult' object does not support item assignment

You should use _replace. See documentation on namedtuple.

Thank you! Checked it out and it is brilliant. Pushing a commit with the fix.

uranusjr · 2023-05-18T08:56:57Z

Airflow connections is unable to parse the uri strings that contain another level of scheme in their uri strings. For example: spark://k8s://100.68.0.1:443?deploy-mode=cluster.

This is not a valid URI to begin with. You should escape the second :// instead.

amoghrajesh · 2023-05-18T09:11:17Z

Airflow connections is unable to parse the uri strings that contain another level of scheme in their uri strings. For example: spark://k8s://100.68.0.1:443?deploy-mode=cluster.

This is not a valid URI to begin with. You should escape the second :// instead.

Oh I wasn't aware of that. Made the changes accordingly

amoghrajesh · 2023-05-22T04:22:36Z

@uranusjr can you take a look at this PR once you have some time? I have addressed your review comments.

uranusjr · 2023-05-22T06:01:08Z

airflow/models/connection.py

@@ -186,16 +186,43 @@ def _normalize_conn_type(conn_type):
            conn_type = conn_type.replace("-", "_")
        return conn_type

+    @staticmethod
+    def split_uri_to_parts(uri):
+        c = uri.count("://")


Is it ever possible for this to return >=2 if the escape syntax is used…? A valid URL can never contain more than one :// sequence.

Good point, since spark://k8s://100.68.0.1:443?deploy-mode=cluster is not a valid URL as you mentioned earlier. I think the logic in this PR is to have a fallback mechanism in case such a scenario occurs.

In normal cases where the format is fine, the logic minus what this PR adds seems to work fine. I think we can do just by adding the cases we have in test_connection.py as far as this PR is concerned. What do you think @uranusjr ?

Yeah makes sense. Also could be a good idea to add something in the Connection form to emit an error if the user puts in an invalud URL. Probably cleaner to do in a separate PR though.

@uranusjr do you mean on the UI? I am not sure if we have a check for that, it would be great to do that though.

As far as this PR is concerned, I will undo all the changes and just add the test cases, that way we have confidence that the actual code for this works.

I think this can be done in ConnectionForm (see WTForms documentation to add validators). We should probably do that on the host field.

Nice. Let me come up with a separate issue and PR for that :)

amoghrajesh · 2023-05-22T08:16:53Z

@uranusjr re worked the PR and added valid examples for the same. I have also changed the PR title.

amoghrajesh · 2023-05-22T10:20:07Z

@potiuk @uranusjr can you help in merging this PR? It has passed the checks

eladkal · 2023-05-22T10:42:09Z

This PR marked as closing for #21638 but it changes only the test.
How can this fix a user bug when just adding more test coverage without fixing anything in the code itself?

amoghrajesh · 2023-05-22T11:05:59Z

@eladkal the issue described in #21638 is not a common use case. This PR adds some more scenarios to the test case, including what is mentioned in the github issue. Maybe we should not mark is as closes in that case. WDYT @eladkal?
Also, new issue to fix this properly is up #31460

hussein-awala · 2023-05-22T11:18:33Z

@amoghrajesh I'll take a look to see how this bug can be fixed

amoghrajesh · 2023-05-22T11:20:18Z

I can work with you on that, this PR should be closed as not closing the issue which is linked.

eladkal · 2023-05-22T11:58:00Z

This PR adds some more scenarios to the test case, including what is mentioned in the github issue.

that is OK just don't mark the PR as closing the issue since it does not

amoghrajesh · 2023-05-22T12:03:42Z

Removed that portion from the description

amoghrajesh · 2023-05-22T18:33:52Z

@uranusjr @potiuk can we merge this PR? Clean build and 2 approvals so far.

hussein-awala · 2023-05-22T18:38:11Z

@amoghrajesh, I'm currently working on the fix. Let's wait before merging this PR to see if we need to add more tests.

amoghrajesh · 2023-05-22T18:41:01Z

@hussein-awala a related issue that might be interesting or help you with the fix: #31460

amoghrajesh · 2023-06-11T11:45:14Z

@hussein-awala can we merge this one in after we merge the dependent PR?

amoghrajesh · 2023-06-24T03:03:34Z

@hussein-awala can we merge this PR too as the dependent PR has been merged recently?

--------- Co-authored-by: Amogh <adesai@cloudera.com>

Issue-21638: Spark Connection with k8s in URL not mapped correctly

ef777d8

amoghrajesh requested review from kaxil, XD-DENG and ashb as code owners May 18, 2023 08:34

uranusjr reviewed May 18, 2023

View reviewed changes

Fixing nits

3a985cd

amoghrajesh requested a review from uranusjr May 18, 2023 09:12

eladkal changed the title ~~Issue-21638: Spark Connection with k8s in URL not mapped correctly~~ Fix Spark Connection with k8s in URL not mapped correctly May 18, 2023

Using _replace for named tuple

ebafb87

uranusjr reviewed May 22, 2023

View reviewed changes

Undoing url split changes

b883531

amoghrajesh changed the title ~~Fix Spark Connection with k8s in URL not mapped correctly~~ Add valid spark on k8s airflow connection urls to tests May 22, 2023

amoghrajesh requested a review from uranusjr May 22, 2023 08:16

uranusjr approved these changes May 22, 2023

View reviewed changes

potiuk approved these changes May 22, 2023

View reviewed changes

amoghrajesh mentioned this pull request May 22, 2023

Add capability in Airflow connections to validate host #31460

Closed

2 tasks

potiuk merged commit e21eae0 into apache:main Jun 24, 2023
42 checks passed

ferruzzi pushed a commit to aws-mwaa/upstream-to-airflow that referenced this pull request Jun 27, 2023

Add valid spark on k8s airflow connection urls to tests (apache#31376)

3f4f8cf

--------- Co-authored-by: Amogh <adesai@cloudera.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add valid spark on k8s airflow connection urls to tests #31376

Add valid spark on k8s airflow connection urls to tests #31376

amoghrajesh commented May 18, 2023 •

edited

uranusjr May 18, 2023

amoghrajesh May 18, 2023

uranusjr May 19, 2023

amoghrajesh May 19, 2023

uranusjr May 19, 2023

amoghrajesh May 19, 2023

uranusjr commented May 18, 2023

amoghrajesh commented May 18, 2023

amoghrajesh commented May 22, 2023

uranusjr May 22, 2023

amoghrajesh May 22, 2023

amoghrajesh May 22, 2023

uranusjr May 22, 2023

amoghrajesh May 22, 2023

amoghrajesh May 22, 2023

uranusjr May 22, 2023

amoghrajesh May 22, 2023

amoghrajesh commented May 22, 2023

amoghrajesh commented May 22, 2023

eladkal commented May 22, 2023

amoghrajesh commented May 22, 2023

hussein-awala commented May 22, 2023

amoghrajesh commented May 22, 2023

eladkal commented May 22, 2023

amoghrajesh commented May 22, 2023

amoghrajesh commented May 22, 2023

hussein-awala commented May 22, 2023

amoghrajesh commented May 22, 2023

amoghrajesh commented Jun 11, 2023

amoghrajesh commented Jun 24, 2023

Add valid spark on k8s airflow connection urls to tests #31376

Add valid spark on k8s airflow connection urls to tests #31376

Conversation

amoghrajesh commented May 18, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

uranusjr commented May 18, 2023

amoghrajesh commented May 18, 2023

amoghrajesh commented May 22, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

amoghrajesh commented May 22, 2023

amoghrajesh commented May 22, 2023

eladkal commented May 22, 2023

amoghrajesh commented May 22, 2023

hussein-awala commented May 22, 2023

amoghrajesh commented May 22, 2023

eladkal commented May 22, 2023

amoghrajesh commented May 22, 2023

amoghrajesh commented May 22, 2023

hussein-awala commented May 22, 2023

amoghrajesh commented May 22, 2023

amoghrajesh commented Jun 11, 2023

amoghrajesh commented Jun 24, 2023

amoghrajesh commented May 18, 2023 •

edited