Test fixing bad PROD image preparation in CI #39011

…on dependencies in pyproject.toml. Signed-off-by: sudipto baral <sudiptobaral.me@gmail.com>

Signed-off-by: sudipto baral <sudiptobaral.me@gmail.com>

…ith 'set'

Signed-off-by: sudipto baral <sudiptobaral.me@gmail.com>

…e errors

Signed-off-by: sudipto baral <sudiptobaral.me@gmail.com>

…ogin to use path instead of status code.

Signed-off-by: sudipto baral <sudiptobaral.me@gmail.com>

There were two problem was with session handling: * the get_session_cookie - did not get the right cookie - it returned "session" string. The right fix was to change cookie_jar into cookie.jar because this is where apparently TestClient of starlette is holding the cookies (visible when you debug) * The client does not accept "set_cookie" method - it accepts passing cookies via "cookies" dictionary - this is the usual httpx client - see https://www.starlette.io/testclient/ - so we have to set cookie directly in the get method to try it out I added few more calls to show what's going on and to see that the call works before logout The other two tests shoudl be fixed similarly.

Some tests require functionality not available to Starlette test client as they use Flask test client specific features - for those we have an option to get flask test client instead of starlette one.

Error handling for Connexion 3 integration needed to be reworked. It's likely not a final version - but the way it behaves is much the same as it works in main: * for API errors - we get application/problem+json responses * for UI erros - we have rendered views * for redirection - we have correct location header (it's been missing) * the api error handled was not added as available middleware in the www tests It should fix all test_views_base.py tests which were failing on lack of location header for redirection.

The problem in the test was that Starlette Test Client opens a new connection and start new session, while flask test client uses the same database session. The test did not show data because the data was not committed and session was not closed - which also failed sqlite local tests with "database is locked" error. This solution can be applied to other tests where data is differnet than expected and there is a missing commit / close when data is prepared.

The tests were failing again because the dagrun created was not committed and session not closed. This worked with flask client that used the same session accidentally but did not work with test client from Starlette. Also it caused "database locked" in sqlite / local tests. There are still two tests failing with different response - to be investigated.

Example fixes to fix tests testing error handling

This PR partially fixes sessions and request parameter for test_views_log. Some tests are still failing but for different reasons - to be investigated.

The problem in those tests was that the check in security manager was based on the assumption that the security manager was shared between the client and test flask application - because they were coming from the same flask app. But when we use starlette, the call goes to a new process started and the user is deleted in the database - so the shortcut of checking the security manager did not work. The change is that we are now checking if the user is deleted by calling /users/show (we need a new users READ permission for that) - this way we go to the database and check if the user was indeed deleted.

Another session (implicit) not committed.

Another cases where sessions were not closed

Signed-off-by: sudipto baral <sudiptobaral.me@gmail.com>

There were two reasons for the test failed: * when the Job was added to task instance, the task instance was not merged in session, which means that commit did not store the added Job * some of the tests were expecting a call with specific session and they failed because session was different. Replacing the session with mock.ANY tells pytest that this parameter can be anything - we will have different session when when the call will be made with ASGI/Starlette

* added default value for limit parameter across the board. Connexion 3 does not like if the parameter had no default and we had not provided one - even if our custom decorated was adding it. Adding default value and updating our decorator to treat None as `default` fixed a number of problems where limits were not passed * swapped openapi specification for /datasets/{uri} and /dataset/events. Since `{uri}` was defined first, connection matched `events` with `{uri}` and chose parameter definitions from `{uri}` not events * few other smaller fixes

* Using flask client helped.

The problem here was that some sessions should be committed/closed but also in order to run it standalone we wanted to create log templates in the database - as it relied implcitly on log templates created by other tests. Also handling of the response without conteent type had to be fixed. Remaining issue is 401 vs 403 forbidden returned. To be looked at later.

Signed-off-by: sudipto baral <sudiptobaral.me@gmail.com>

Switching to flask client rather than starlette, helped to fix the issue.

The fix checks for the 429 HTTP exception that should be returned in this case. This also reverts commit a9aa27d.

Fixed by switching to use flask client for testing rather than starlette. Starlette client in this case has some side effects that are also impacting Sqlite's session being created in a different thread and deleted with close_all_sessions fixture.

Fixed by switching the test to flask_admin_client. Removes sqlalchemy session creted from a different thread.

When PROD image packages are installed in in CI, the local sources should not be present in the image, also constraints from sources shoudl replace the one downloaded from main.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test fixing bad PROD image preparation in CI #39011

Test fixing bad PROD image preparation in CI #39011

Commits on Apr 14, 2024

Commits on Apr 15, 2024