Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate to connexion v3 #37638

Closed
wants to merge 109 commits into from

Conversation

sudiptob2
Copy link
Contributor

@sudiptob2 sudiptob2 commented Feb 22, 2024

This PR is created based on #36052

Todo

  • Taks 1 - Refactor get_api_endpoints() as it can not return blueprint in connexion v3
  • Taks 2 - Find a way to replace the Legacy environ_overrides method from the old version of Connexion to fix the problem in tests.
  • Task 3 - Fix unit tests that use app

Taks 1 - Refactor get_api_endpoints()

Problem Definiton

Ref: Github Pull Request #36052 VladaZakharova commented on Jan 18

In the init_api_auth_provider method, we update the base path as follows:

def init_api_auth_provider(connexion_app: connexion.FlaskApp):
    """Initialize the API offered by the auth manager."""
    auth_mgr = get_auth_manager()
    blueprint = auth_mgr.get_api_endpoints(connexion_app)
    if blueprint:
        base_paths.append(blueprint.url_prefix if blueprint.url_prefix else "")
        flask_app = connexion_app.app
        flask_app.extensions["csrf"].exempt(blueprint)

However, the blueprint object obtained from auth_mgr.get_api_endpoints(connexion_app) will always be None if we are using ConnexionV3.

Proposed solution

Ref vincbeck commented on Jan 18

  • Rename get_api_endpoints to set_api_endpoints. The return type should be updated to None. Documentation should be updated as well to something like "Set API endpoint(s) definition for the auth manager.". This is a breaking change but nobody uses this interface yet, so it is a good time to do it.
  • This piece of code flask_app.extensions["csrf"].exempt(blueprint) should be moved in the set_api_endpoints method using appbuilder.app.extensions["csrf"].exempt(api.blueprint) Ref: Migrate to connexion v3 #37638 (comment)

How to test

  • Run client tests python ./clients/python/test_python_client.py

Subtasks

Task 2 - Replace envrion_overrides argument

Problem Definition

Ref: Github Pull Request #36052 commented on Feb 6

Find a way to replace legacy environ_overrides method from old version of Connexion to fix the problem in tests
TypeError: get() got an unexpected keyword argument 'environ_overrides'. Contact @RobbeSneyders for some help. Currently we didn't find anything in their documentation regarding new way how to override env variables.

Solution

  • Use headers to send {'REMOTE_USER:"user"} instead of using environ_overirdes argument inside testclient.method. Accordinly update the authentication part in tests/test_utils/remote_user_api_auth_backend.py to user_id = request.headers.get("REMOTE_USER")

@boring-cyborg boring-cyborg bot added area:API Airflow's REST/HTTP API area:CLI area:providers area:webserver Webserver related Issues provider:fab labels Feb 22, 2024
@sudiptob2 sudiptob2 force-pushed the feat/migrate-to-connexion-v3 branch 2 times, most recently from 3013d04 to db5fef3 Compare February 22, 2024 22:38
@sudiptob2
Copy link
Contributor Author

Hi @vincbeck

We have defined the scope of Task 1 in the PR description. Thank you for your previous comments and suggestions to complete this task. There is a part of the proposed solution that we are still unclear.
Refferening to VladaZakharova commented 2 weeks ago

Blueprint is now can't be retrieved from connexion app and there will be always None.
Find a way to add csrf extension to a newly created blueprint using connexion: to retrieve blueprint object from connexion_app variable to save the current logic (flask_app.extensions["csrf"].exempt(blueprint)) or find a way to add this extension on connexion level(check the documentation for available options).

In a recent comment RobbeSneyders commented 2 days ago suggested

The most future-proof solution would be to move the csfr protection to a middleware such as this one. The skif_if_scope parameter seems to be a replacement for the exempt functionality you're using.

So, should we still consider moving flask_app.extensions["csrf"].exempt(blueprint) in the set_api_endpoints method using appbuilder.app.extensions["csrf"].exempt(api.blueprint) ?

@vincbeck
Copy link
Contributor

Hi @vincbeck

We have defined the scope of Task 1 in the PR description. Thank you for your previous comments and suggestions to complete this task. There is a part of the proposed solution that we are still unclear. Refferening to VladaZakharova commented 2 weeks ago

Blueprint is now can't be retrieved from connexion app and there will be always None.
Find a way to add csrf extension to a newly created blueprint using connexion: to retrieve blueprint object from connexion_app variable to save the current logic (flask_app.extensions["csrf"].exempt(blueprint)) or find a way to add this extension on connexion level(check the documentation for available options).

In a recent comment RobbeSneyders commented 2 days ago suggested

The most future-proof solution would be to move the csfr protection to a middleware such as this one. The skif_if_scope parameter seems to be a replacement for the exempt functionality you're using.

So, should we still consider moving flask_app.extensions["csrf"].exempt(blueprint) in the set_api_endpoints method using appbuilder.app.extensions["csrf"].exempt(api.blueprint) ?

Yes

@sudiptob2
Copy link
Contributor Author

Refinement: Returning flask_app instead of connexion_app

Ref: #37555 (comment) Mentions: @vincbeck

We investigated the usage of create_app() method and found a few places where we need to call connexion_app.run()
If we return flask_app from the create_app() method then maybe it is not possible to handle those use cases. Therefore, we should return connexion_app. There might be other ways to handle this, let me know your opinion.
Usage 1:

app = create_app(testing=conf.getboolean("core", "unit_test_mode"))
app.run(
debug=True,

Usage 2:

app = create_app(testing=conf.getboolean("core", "unit_test_mode"))
app.run(
debug=True, # nosec

@sudiptob2 sudiptob2 force-pushed the feat/migrate-to-connexion-v3 branch 5 times, most recently from f66f6d9 to de01bb4 Compare February 24, 2024 15:29
@Satoshi-Sh
Copy link
Contributor

Satoshi-Sh commented Feb 24, 2024

Problem

Due to the upgrade of connextion v3, we cannot access blueprints( they moved the blueprint registration code inside their codebase). We used the returned blueprint to make exemptions to accept HTTP(S) requests without "csrf token" in the header. When the auth-token is in the header, the client doesn't include a csrf token. That's why we get csrf token missing error with test_python_client.py

@RobbeSneyders suggested utilizing the middleware library asgi-csrf to do the same without using blueprints.

def skip_api_paths(scope)
    return scope["path"].startswith("/api/")

app = asgi_csrf(
    app,
    signing_secret="secret-goes-here",
    skip_if_scope=skip_api_paths
)

This is a sample code to make csrf-token exemption.

  • What the score will be in the airflow project?
  • I'm not sure about siginig_secret. Do I need to generate it or can get it from somewhere?

@potiuk
Copy link
Member

potiuk commented Feb 25, 2024

What the score will be in the airflow project?

I assume the scope :) ?. I believe the scope is the base URL of Airflow webserver (not 100% sure how asg-csrf does it but that's what I understand it should be. The CSRF tokens we have are generated in the webserver views - and those are generated at the "base URL" (and anything that's deeper in the path) - and those csrf tokens are then used by the browser to make the calls to the API.

I'm not sure about siginig_secret. Do I need to generate it or can get it from somewhere?

We should use https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#secret-key - this is done usually by:

conf.get_mandatory_value("webserver", "secret_key")

@Satoshi-Sh
Copy link
Contributor

Satoshi-Sh commented Feb 25, 2024

Yes, it's scope. I have this now.

# asgi-csrf skip_if_scope
    def skip_api_paths(scope):
        return scope["path"].startswith("/api/v1")

    flask_app = asgi_csrf(
        flask_app,
        signing_secret=conf.get_mandatory_value("webserver", "secret_key"),
        skip_if_scope=skip_api_paths,
    )


After using asgi_csrf, I get a different error.
RuntimeError: A secret key is required to use CSRF. INFO: "POST /api/v1/dags/exmple_bash_operator/dagRuns HTTP/1.1" instead of something like CSRF token is missing on the same endpoint.

I hope I'm in the right direction to solve the issue. Here is my pull request to Sudipto's forked repo.

@sudiptob2
Copy link
Contributor Author

Yes, it's scope. I have this now.

# asgi-csrf skip_if_scope
    def skip_api_paths(scope):
        return scope["path"].startswith("/api/v1")

    flask_app = asgi_csrf(
        flask_app,
        signing_secret=conf.get_mandatory_value("webserver", "secret_key"),
        skip_if_scope=skip_api_paths,
    )

After using asgi_csrf, I get a different error. RuntimeError: A secret key is required to use CSRF. INFO: "POST /api/v1/dags/exmple_bash_operator/dagRuns HTTP/1.1" instead of something like CSRF token is missing on the same endpoint.

I hope I'm in the right direction to solve the issue. Here is my pull request to Sudipto's forked repo.

It is possible to avoid this error using the following tweak for the time being. asgi-csrf looks for SECRET_KEY variable to be set. But it still does not solve the problem of missing CSRF token. In my opinion, it wont be this simple to add asgi_csrf middleware.

    # asgi-csrf skip_if_scope
    flask_app.config['SECRET_KEY'] = conf.get_mandatory_value("webserver", "secret_key")

    def skip_api_paths(scope):
        return scope["path"].startswith("/api/v1")

    asgi_csrf(
        flask_app,
        signing_secret=conf.get_mandatory_value("webserver", "secret_key"),
        skip_if_scope=skip_api_paths,
    )

@potiuk
Can we get a demo/explanation session in the next meeting regarding how the CSRF protection works in airflow, especially in the context of this PR?

@sudiptob2
Copy link
Contributor Author

sudiptob2 commented Feb 25, 2024

We might be able to handle csrf exemption logic in the following way.

    @connexion_app.app.before_request
    def before_request():
        """Exempts the view function associated with '/api/v1' requests from CSRF protection."""
        if request.path.startswith("/api/v1"):  # TODO: make sure this path is correct
            view_function = flask_app.view_functions.get(request.endpoint)
            if view_function:
                # Exempt the view function from CSRF protection
                connexion_app.app.extensions["csrf"].exempt(view_function)

I implemented it here, asking for a review @vincbeck @potiuk @Satoshi-Sh

@Satoshi-Sh
Copy link
Contributor

Nicely done, @sudiptob2 . I checked it with test_python_client.py. The first error is the same as before this update and the second one is just error 404 instead of the csrf issues. I'm not sure if this has something to do with our missing base_paths.append(blueprint.url_prefix if blueprint.url_prefix else "") , or maybe it's related to issue 2 or something else.

Once todo about the scope is done, we can go ahead to the second bug.

Screenshot from 2024-02-25 14-37-29

@potiuk potiuk requested a review from vincbeck February 25, 2024 21:17
@potiuk
Copy link
Member

potiuk commented Feb 25, 2024

cc: @VladaZakharova - just adding you for awareness :)

@potiuk
Copy link
Member

potiuk commented Feb 25, 2024

Can we get a demo/explanation session in the next meeting regarding how the CSRF protection works in airflow, especially in the context of this PR?

Once I do a bit of homework on it myself :)

@sudiptob2
Copy link
Contributor Author

sudiptob2 commented Feb 25, 2024

Nicely done, @sudiptob2 . I checked it with test_python_client.py. The first error is the same as before this update and the second one is just error 404 instead of the csrf issues. I'm not sure if this has something to do with our missing base_paths.append(blueprint.url_prefix if blueprint.url_prefix else "") , or maybe it's related to issue 2 or something else.

Once todo about the scope is done, we can go ahead to the second bug.

Screenshot from 2024-02-25 14-37-29

Hi @Satoshi-Sh,
Did you load example dags while testing? You can use the following command to runing breeze.

breeze start-airflow --dev-mode --load-example-dags --backend postgres

base_paths.append(blueprint.url_prefix if blueprint.url_prefix else "") , or maybe it's related to issue 2 or something else.

This has to be handled in subtask 1 so that reviewers can easily review it.

@potiuk
Copy link
Member

potiuk commented Apr 16, 2024

Hi @potiuk. Great to hear that you found the issue. Let me know if there is anything else I can help with.

Hey @RobbeSneyders Thanks for the offer. We got the PR green finally (HURRAY!). What - I think, you could help with is to validate some of our assumptions. Due to the length of this PR and comments and number of commits in that, I will open a new PR and ask some concrete questions and explain our decisions there and ask you for comments - and I will involve other maintainers as well.

potiuk pushed a commit to potiuk/airflow that referenced this pull request Apr 16, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>
@potiuk
Copy link
Member

potiuk commented Apr 16, 2024

Continued in #39055

potiuk pushed a commit to potiuk/airflow that referenced this pull request Apr 16, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>
potiuk pushed a commit to potiuk/airflow that referenced this pull request Apr 20, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>
potiuk pushed a commit to potiuk/airflow that referenced this pull request Apr 20, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>
potiuk pushed a commit to potiuk/airflow that referenced this pull request Apr 24, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>
potiuk pushed a commit to potiuk/airflow that referenced this pull request Apr 25, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>
potiuk pushed a commit to potiuk/airflow that referenced this pull request Apr 26, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>
potiuk pushed a commit to potiuk/airflow that referenced this pull request May 7, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>
potiuk pushed a commit to potiuk/airflow that referenced this pull request May 7, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>
potiuk pushed a commit to potiuk/airflow that referenced this pull request May 26, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>
potiuk pushed a commit to potiuk/airflow that referenced this pull request May 26, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>
potiuk pushed a commit to potiuk/airflow that referenced this pull request May 28, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>

Better API initialization including vending of API specification.

The way paths are added and initialized is better (for example
FAB contributes their path via new method in Auth Manager.

This also add back-compatibility to FAB auth manaager to continue
working on Airflow 2.9.
potiuk pushed a commit to potiuk/airflow that referenced this pull request May 28, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>

Better API initialization including vending of API specification.

The way paths are added and initialized is better (for example
FAB contributes their path via new method in Auth Manager.

This also add back-compatibility to FAB auth manaager to continue
working on Airflow 2.9.
potiuk pushed a commit to potiuk/airflow that referenced this pull request May 28, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>

Better API initialization including vending of API specification.

The way paths are added and initialized is better (for example
FAB contributes their path via new method in Auth Manager.

This also add back-compatibility to FAB auth manaager to continue
working on Airflow 2.9.
potiuk pushed a commit to potiuk/airflow that referenced this pull request May 28, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>

Better API initialization including vending of API specification.

The way paths are added and initialized is better (for example
FAB contributes their path via new method in Auth Manager.

This also add back-compatibility to FAB auth manaager to continue
working on Airflow 2.9.
potiuk pushed a commit to potiuk/airflow that referenced this pull request May 29, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>

Better API initialization including vending of API specification.

The way paths are added and initialized is better (for example
FAB contributes their path via new method in Auth Manager.

This also add back-compatibility to FAB auth manaager to continue
working on Airflow 2.9.
potiuk pushed a commit to potiuk/airflow that referenced this pull request May 29, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>

Better API initialization including vending of API specification.

The way paths are added and initialized is better (for example
FAB contributes their path via new method in Auth Manager.

This also add back-compatibility to FAB auth manaager to continue
working on Airflow 2.9.
potiuk pushed a commit to potiuk/airflow that referenced this pull request May 29, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>

Better API initialization including vending of API specification.

The way paths are added and initialized is better (for example
FAB contributes their path via new method in Auth Manager.

This also add back-compatibility to FAB auth manaager to continue
working on Airflow 2.9.
potiuk pushed a commit to potiuk/airflow that referenced this pull request May 29, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>

Better API initialization including vending of API specification.

The way paths are added and initialized is better (for example
FAB contributes their path via new method in Auth Manager.

This also add back-compatibility to FAB auth manaager to continue
working on Airflow 2.9.
potiuk pushed a commit to potiuk/airflow that referenced this pull request Jun 24, 2024
This is a huge PR being result of over a 100 commits
made by a number of people in #apache#36052 and apache#37638. It
switches to Connexion 3 as the driving backend
implementation for both - Airflow REST APIs and Flask
app that powers Airflow UI. It should be largely
backwards compatible when it comes to behaviour of
both APIs and Airflow Webserver views, however due to
decisions made by Connexion 3 maintainers, it changes
heavily the technology stack used under-the-hood:

1) Connexion App is an ASGI-compatible Open-API spec-first
   framework using ASGI as an interface between webserver
   and Python web application. ASGI is an asynchronous
   successor of WSGI.

2) Connexion itself is using Starlette to run asynchronous
   web services in Python.

3) We continue using gunicorn appliation server that still
   uses WSGI standard, which means that we can continue using
   Flask and we are usig standard Uvicorn ASGI webserver that
   converts the ASGI interface to WSGI interface of Gunicorn

Some of the problems handled in this PR

There were two problem was with session handling:

* the get_session_cookie - did not get the right cookie - it returned
  "session" string. The right fix was to change cookie_jar into
  cookie.jar because this is where apparently TestClient of starlette
  is holding the cookies (visible when you debug)

* The client does not accept "set_cookie" method - it accepts passing
  cookies via "cookies" dictionary - this is the usual httpx client
  - see  https://www.starlette.io/testclient/ - so we have to set
  cookie directly in the get method to try it out

Add "flask_client_with_login" for tests that neeed flask client

Some tests require functionality not available to Starlette test
client as they use Flask test client specific features - for those
we have an option to get flask test client instead of starlette
one.

Fix error handling for new connection 3 approach

Error handling for Connexion 3 integration needed to be reworked.

The way it behaves is much the same as it works in main:

* for API errors - we get application/problem+json responses
* for UI erros - we have rendered views
* for redirection - we have correct location header (it's been
  missing)
* the api error handled was not added as available middleware
  in the www tests

It should fix all test_views_base.py tests which were failing
on lack of location header for redirection.

Fix wrong response is tests_view_cluster_activity

The problem in the test was that Starlette Test Client opens a new
connection and start new session, while flask test client
uses the same database session. The test did not show data because
the data was not committed and session was not closed - which also
failed sqlite local tests with "database is locked" error.

Fix test_extra_links

The tests were failing again because the dagrun created was not
committed and session not closed. This worked with flask client
that used the same session accidentally but did not work with
test client from Starlette. Also it caused "database locked"
in sqlite / local tests.

Switch to non-deprecated auth manager

Fix to test_views_log.py

This PR partially fixes sessions and request parameter for
test_views_log. Some tests are still failing but for different reasons -
to be investigated.

Fix views_custom_user_views tests

The problem in those tests was that the check in security manager
was based on the assumption that the security manager was shared
between the client and test flask application - because they
were coming from the same flask app. But when we use starlette,
the call goes to a new process started and the user is deleted in
the database - so the shortcut of checking the security manager
did not work.

The change is that we are now checking if the user is deleted by
calling /users/show (we need a new users READ permission for that)
 - this way we go to the database and check if the user was indeed
deleted.

Fix test_task_instance_endpoint tests

There were two reasons for the test failed:

* when the Job was added to task instance, the task instance was not
  merged in session, which means that commit did not store the added
  Job

* some of the tests were expecting a call with specific session
  and they failed because session was different. Replacing the
  session with mock.ANY tells pytest that this parameter can be
  anything - we will have different session when when the call will
  be made with ASGI/Starlette

Fix parameter validation

* added default value for limit parameter across the board. Connexion
  3 does not like if the parameter had no default and we had not
    provided one - even if our custom decorated was adding it. Adding
  default value and updating our decorator to treat None as `default`
  fixed a number of problems where limits were not passed

* swapped openapi specification for /datasets/{uri} and /dataset/events.
  Since `{uri}` was defined first, connection matched `events` with
  `{uri}` and chose parameter definitions from `{uri}` not events

Fix test_log_enpoint tests

The problem here was that some sessions should be committed/closed but
also in order to run it standalone we wanted to create log templates
in the database - as it relied implcitly on log templates created by
other tests.

Fix test_views_dagrun, test_views_tasks and test_views_log

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Fix test_views_dagrun

Fixed by switching to use flask client for testing rather than
starlette. Starlette client in this case has some side effects that are
also impacting Sqlite's session being created in a different
thread and deleted with close_all_sessions fixture.

Co-authored-by: sudipto baral <sudiptobaral.me@gmail.com>
Co-authored-by: satoshi-sh <satoss1108@gmail.com>
Co-authored-by: Maksim Yermakou <maksimy@google.com>
Co-authored-by: Ulada Zakharava <Vlada_Zakharava@epam.com>

Better API initialization including vending of API specification.

The way paths are added and initialized is better (for example
FAB contributes their path via new method in Auth Manager.

This also add back-compatibility to FAB auth manaager to continue
working on Airflow 2.9.
Copy link

github-actions bot commented Aug 3, 2024

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Aug 3, 2024
@github-actions github-actions bot closed this Aug 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:API Airflow's REST/HTTP API area:CLI area:providers area:webserver Webserver related Issues default versions only When assigned to PR - only default python version is used for CI tests provider:fab stale Stale PRs per the .github/workflows/stale.yml policy file
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants