Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Properly handle multiple requests to threaded REST API #3974

Merged
merged 2 commits into from
May 11, 2020
Merged

Properly handle multiple requests to threaded REST API #3974

merged 2 commits into from
May 11, 2020

Commits on May 11, 2020

  1. Add possibility to pass keyword arguments to create_engine 

    The function `aiida.backends.utils.create_sqlalchemy_engine` now takes
keyword arguments that are passed straight to the SQLAlchemy function
`create_engine` which creates the engine that connects to the database.
This utility function is used by the SQLAlchemy and by both backends
for the `QueryBuilder`. By allowing these keyword arguments to be passed
and plumbing them all the way in to the `Backend` class, the parameters
of the SQLAlchemy engine and the associated queue pool can be
customized. This is not meant to be used by end users, but can be
important for important use, for testing and performance reasons.
    @CasperWA @sphuber
    CasperWA authored and sphuber committed May 11, 2020
    Configuration menu
    Copy the full SHA
    177c353 View commit details
    Browse the repository at this point in the history

  2. Close SQLA session after every REST API request 

    This is needed due to the server running in threaded mode, i.e.,
creating a new thread for each incoming request. This concept is great
for handling many requests, but crashes when used together with AiiDA's
global singleton SQLA session used, no matter the backend of the profile
by the `QueryBuilder`.

Specifically, this leads to issues with the SQLA QueuePool, since the
connections are not properly released when a thread is closed. This
leads to unintended QueuePool overflow.

This fix wraps all HTTP method requests and makes sure to close the
current thread's SQLA session after the request as been completely
handled.

Use Flask-RESTful's integrated `Resource` attribute `method_decorators`
to apply `close_session` wrapper to all and any HTTP request that may
be requested of AiiDA's `BaseResource` (and its sub-classes).

Additionally, remove the `__init__` function overwritten in
`Node(BaseResource)`, since it is redundant, and the attributes `tclass`
is not relevant with v4 (AiiDA v1.0.0 and above), but was never removed.
It should have been removed when moving to v4 in 4ff2829.

Concerning the added tests: the timeout needs to be set for Python 3.5
in order to stop the http socket and properly raise (and escape out of
an infinite loop). The `capfd` fixture must be used, otherwise the
exception cannot be properly captured.

The tests were simplified into the pytest scheme with ideas from
@sphuber and @greschd.
    @CasperWA @sphuber
    CasperWA authored and sphuber committed May 11, 2020
    Configuration menu
    Copy the full SHA
    11130a4 View commit details
    Browse the repository at this point in the history