task/WG-141: Fix db connections for celery workers

[nathanfranklin]

Overview:

Disable pool connections for celery workers. This is a temp fix/workaround for our connection related issues.

This PR refactors how database session is used in celery worker tasks. Now, create_task_session is a context manger that provides a session and then ensure that session is closed after its used (and rolled-back if there was an exception).

Tasks look like the following:

@app.task()
def some_task():
    """
    Refresh all observable projects
    """
    with create_task_session() as session:
        # use the session
         point_cloud = PointCloudService.get(session, pointCloudId)

    # `session.close` is called by the context manager
    # if there is an unhandled exception then a`session.rollback()` is also performed.

So going forward,db_session from db.py is a scoped_session that is used when processing requests in Flask and also in unit tests for fixtures and asserts. Then for all celery tasks, the create_task_session is used.

PR Status:

• Ready.

Related Jira tickets:

• WG-141

[nathanfranklin]

FYI: not directly related to the changes in this PR, but this is how the db sessions are ensured to be closed properly after each request. I'm not certain but something similar in celery task might be needed.

[sophia-massie]

LGTM

[nathanfranklin]

This is the only large out-of-scope change. I drop this endpoint which allows point clouds to be directly uploaded in a POST. This endpoint is NOT being used by frontend. Instead of uploading a file,the current frontends allow users to select to import an existing pointcloud on DesignSafe and then the backend gets the file on their behalf and processes it.

We have some other non-used endpoints where users can upload file directly but none of the frontends use those anymore. All the frontends use the endpoint where users say which file from DS Tapis we should import for them.

Really, we should probably drop all the upload routs.

I'm dropping this one here as it made refactoring the use of database sessions by point cloud tasks and methods MUCH easier.

[sophia-massie]

LGTM! Thanks for the detail on the tests.

[taoteg]

LGTM (retroactively).