Set up CI API test framework to check database state

[svenseeberg]

Motivation

In our CI API tests we only check for response codes and return values. In some cases, for example feedback, we have currently have no way of verifying if the feedback has actually been saved to the database.

Proposed Solution

Add a new parameter to to our API (and potentially other tests) that allows checking the state of the database. One option for implementing this would be to have two additional parameters in our list of tests:

• A raw(?) SQL query that is run
• An expected output/return value of the query

Alternatives

Find better alternatives to check if a database manipulation was actually successful.

Additional Context

Design Requirements

[timobrembeck]

I think for all those views, the one-fits-all approach doesn't quite work.
Instead, I suggest to add single test functions where not only the database state can be tested, but also the return html can be inspected for e.g. the error/success messages, the logging output can be traced, different form inputs can be checked to ensure that invalid input correctly throws an error etc... similar tests which are already implemented are e.g.:

• Test PDF export: https://github.com/digitalfabrik/integreat-cms/blob/develop/tests/pdf/test_pdf_export.py
• Test page filters: https://github.com/digitalfabrik/integreat-cms/blob/develop/tests/cms/test_page_filters.py
• Text XLIFF import/export: https://github.com/digitalfabrik/integreat-cms/blob/develop/tests/xliff/xliff_test.py
• Test region duplication: https://github.com/digitalfabrik/integreat-cms/blob/develop/tests/cms/test_duplicate_regions.py

[svenseeberg]

I agree that there there is not one solution for all tests. But I think for the API calls we could extend https://github.com/digitalfabrik/integreat-cms/blob/develop/tests/api/api_config.py#L6 additional parameters to check the database state. That would potentially allow us to merge the Push Content API tests with the other API tests. Alternatively, we could split between read only and writable API endpoints?

In regards to front end view I think you're totally right. These are too diverse for reducing this to a common list of parameters.

I'll adjust the title of the issue.

[seluianova]

Find better alternatives to check if a database manipulation was actually successful.

I would suggest checking changes through models. I.e. query data into a model object and validate its fields.

[seluianova]

Added checks for cms_feedback table. Also need to cover tables for different feedback types, like cms_pagefeedback, cms_poifeedback etc.

[seluianova]

Added checks for cms_feedback table. Also need to cover tables for different feedback types, like cms_pagefeedback, cms_poifeedback etc.

I think an existing type check might be actually enough here...

assert isinstance(feedback, feedback_type_dict[view_name])