Improve REST API testing system #4403

sizov-kirill · 2022-03-01T11:00:24Z

Motivation and context

The current test database initialization strategy for REST API tests uses the cvat_db.sql file, this approach makes it difficult to develop tests in parallel because it is not clear how to perform *.sql file merging. So this PR solve this problem with Django functionality.

This PR suggest to initialize test db in this way:

#Dump
docker exec cvat python manage.py dumpdata --indent 2 >  tests/rest_api/assets/cvat_db/data.json

#Restore
docker container cp tests/rest_api/assets/cvat_db/data.json cvat:data.json
docker exec cvat python manage.py loaddata /data.json

Increase time with new recovery strategy: ~4 sec.

Change confest.py according new approach of initializing test db.
Update JSON assets
Update tests according to new changes
Estimate convenience of merging two branches that update test db.
Reorganize assets directory
Delete all things that relates with old testing system
Update README

How has this been tested?

Checklist

I submit my changes into the develop branch
I have added a description of my changes into CHANGELOG file
I have updated the documentation accordingly
I have added tests to cover my changes
I have linked related issues (read github docs)
I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

License

I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
I have updated the license header for each file (see an example below)

# Copyright (C) 2022 Intel Corporation
#
# SPDX-License-Identifier: MIT

nmanovic · 2022-03-09T17:11:06Z

tests/rest_api/conftest.py

-    cvat_db_container('psql -U root -q -d test_db -f /cvat_db/cvat_db.sql')
+    _run(f"docker container cp {osp.join(CVAT_DB_DIR, 'restore.sh')} cvat_db:restore.sh")
+    _run(f"docker container cp {osp.join(CVAT_DB_DIR, 'data.json')} cvat:data.json")
+    _run('docker exec cvat python manage.py loaddata /data.json')


psql command supports -v option which allows you to set a variable. Thus you can write cvat_db.sql script with :src_db and :dst_db and set them in cvat_db.sql script. Thus you don't need restore.sh script. More details here: https://stackoverflow.com/questions/36959/how-do-you-use-script-variables-in-psql

nmanovic · 2022-03-09T17:12:35Z

tests/rest_api/assets/cvat_db/restore.sh

@@ -0,0 +1,5 @@
+restore="SELECT pg_terminate_backend(pg_stat_activity.pid) FROM pg_stat_activity WHERE pg_stat_activity.datname = 'cvat' AND pid <> pg_backend_pid();\


It should be replaced by restore_db.sql with variables (see psql -v)

nmanovic · 2022-03-09T17:35:24Z

If I run tests on a non-empty CVAT instance, it will fail. What do you recommend?

(cvat) nmanovic@nmanovic-pc:~/Workspace/cvat$ pytest tests/rest_api/ -s
====================================================== test session starts =======================================================
platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: /mnt/data.ssd1tb/workspace/cvat
collected 162 items                                                                                                              

tests/rest_api/test_analytics.py Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "authtoken_token_user_id_key"
DETAIL:  Key (user_id)=(10) already exists.


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "manage.py", line 21, in <module>
    execute_from_command_line(sys.argv)
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/__init__.py", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/base.py", line 354, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/base.py", line 398, in execute
    output = self.handle(*args, **options)
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 78, in handle
    self.loaddata(fixture_labels)
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 123, in loaddata
    self.load_label(fixture_label)
  File "/opt/venv/lib/python3.8/site-packages/django/core/management/commands/loaddata.py", line 190, in load_label
    obj.save(using=self.using)
  File "/opt/venv/lib/python3.8/site-packages/django/core/serializers/base.py", line 223, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/base.py", line 776, in save_base
    updated = self._save_table(
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/base.py", line 881, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/base.py", line 919, in _do_insert
    return manager._insert(
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/query.py", line 1270, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "/opt/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1416, in execute_sql
    cursor.execute(sql, params)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "/opt/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: Problem installing fixture '/data.json': Could not load authtoken.Token(pk=3952f1aea900fc3daa269473a71c41fac08858b5): duplicate key value violates unique constraint "authtoken_token_user_id_key"
DETAIL:  Key (user_id)=(10) already exists.



! _pytest.outcomes.Exit: Command failed: docker exec cvat python manage.py loaddata /data.json. Add `-s` option to see more details !

nmanovic · 2022-03-09T17:49:30Z

@kirill-sizov , I have a couple of questions:

The PR remotes some directories from cvat_data (e.g., tasks/*). Are you sure that we don't need them? In general a task directory can have raw data, original data, compressed data and some auxiliary files.
How does unziping cvat_data help to resolve conflicts? If somebody creates a task in two parallel PRs, they will have the same primary key (for data, task and jobs). In this case it will be necessary to resolve the conflict manually. Am I right? If so, will it be better to have cvat_data as an archive?

sizov-kirill · 2022-03-10T09:02:13Z

@kirill-sizov , I have a couple of questions:

The PR remotes some directories from cvat_data (e.g., tasks/*). Are you sure that we don't need them? In general a task directory can have raw data, original data, compressed data and some auxiliary files.

How does unziping cvat_data help to resolve conflicts? If somebody creates a task in two parallel PRs, they will have the same primary key (for data, task and jobs). In this case it will be necessary to resolve the conflict manually. Am I right? If so, will it be better to have cvat_data as an archive?

No problem to store other directories if needed, but directory data is enough for now.
You right, the conflict that include same ids should be fixed manually. Storing in unzipped format allows easy to see what changes was maid in commit.

sizov-kirill · 2022-03-10T10:00:36Z

@nmanovic

psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "authtoken_token_user_id_key"
DETAIL:  Key (user_id)=(10) already exists.

I don't exactly know the cause of this problem, but in my case this problem only occurs if my current database was built not with loaddata. To resolve this problem, you can recreate your database:

docker exec cvat_db psql -d postgres -v from:test_db to:cvat -f restore.sql
docker exec cvat_db createdb cvat
docker exec cvat python manage.py migrate

nmanovic · 2022-03-10T16:10:57Z

@nmanovic
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "authtoken_token_user_id_key"
DETAIL:  Key (user_id)=(10) already exists.
I don't exactly know the cause of this problem, but in my case this problem only occurs if my current database was not built with loaddata. To resolve this problem, you can recreate your database:
docker exec cvat_db psql -d postgres -v from:test_db to:cvat -f restore.sql
docker exec cvat_db createdb cvat
docker exec cvat python manage.py migrate

I believe it is because I already have cvat db and it isn't empty. Probably we need to run tests on a clear CVAT instance. There are two ways: say about that in documentation, drop cvat db before start. Need to choose the best solution.

sizov-kirill · 2022-03-14T10:57:57Z

@nmanovic

I decided to keep tar format for data volumes. But we can easily to change this in future, if we will find out some new reasons for storing data volumes in unzipped format.
The problem with exists key error do not occurs in new testing system. This means if your database was create by loaddata then tests will run successfully. For other case need to re-create database as it mentioned in README. If this error will occurs systematically in different cases, I will fix this problem with drop current database before running tests.

nmanovic · 2022-03-14T11:32:52Z

@kirill-sizov , I have merged the PR from Andrey. Could you please resolve conflicts?

sizov-kirill · 2022-03-14T14:52:34Z

@kirill-sizov , I have merged the PR from Andrey. Could you please resolve conflicts?

Done

nmanovic · 2022-03-14T20:08:49Z

tests/rest_api/README.md

-cat assets/cvat_db/cvat_db.sql | docker exec -i cvat_db psql -U root -d cvat
-cat assets/cvat_data.tar.bz2 | docker run --rm -i --volumes-from cvat ubuntu tar -xj --strip 3 -C /home/django/data
+docker container cp assets/cvat_db/data.json cvat:data.json
+docker exec cvat python manage.py loaddata /data.json


Why don't read from stdin? https://docs.djangoproject.com/en/4.0/ref/django-admin/#loading-fixtures-from-stdin

nmanovic · 2022-03-14T20:09:55Z

tests/rest_api/README.md

+  - `cvat_data.tar.bz2` --- archieve with data volumes;
+  - `data.json` --- file required for DB restoring.
+    Contains all information about test db;
+  - `restore.sh` --- simple bash script for creating copy of database and


Do we still have restore.sh?

nmanovic · 2022-03-14T20:11:16Z

tests/rest_api/README.md

+In this case you should terminate all existent connections for cvat database,
+you can perform it with command:
+```
+docker exec cvat_db sh restore.sh cvat test_db


Do we have restore.sh?

nmanovic · 2022-03-14T20:12:40Z

@kirill-sizov , I see cvat_data.tar.bz2 twice in the PR.

nmanovic · 2022-03-14T20:16:21Z

@kirill-sizov , could you please double check your PR next time?

sizov-kirill · 2022-03-15T10:03:46Z

@kirill-sizov , I see cvat_data.tar.bz2 twice in the PR.

I see only one cvat_data.tar.bz2:

tree tests/rest_api/ -P *tar.bz2
tests/rest_api/
├── assets
│   └── cvat_db
│       └── cvat_data.tar.bz2
└── utils

kirill.sizov added 3 commits March 1, 2022 13:46

Models: correct parsing point values from JSON type string

8e660af

Create test db with loaddata functionality

7ed7080

Update JSON assets

062159d

sizov-kirill requested review from azhavoro and nmanovic as code owners March 1, 2022 11:00

Test Jobs: add order independence for shapes

655f9de

sizov-kirill changed the title ~~[WIP] Improve REST API testing system~~ Improve REST API testing system Mar 2, 2022

Conftest: close contections for cvat db before creating test db

de70fde

sizov-kirill changed the title ~~Improve REST API testing system~~ [WIP] Improve REST API testing system Mar 2, 2022

kirill.sizov added 19 commits March 3, 2022 12:42

Pretty JSON format for data.json

7b08c12

Delete sql dump file

d2452a2

Reorganize assets directory

4ab4ccc

Update README

1548df1

Remove 1 space

5e1e741

README: fix typo

4b9541a

Conftest: add immediate exit on any error during db restoring

22cf2cf

cvat_data: delete cache and logs

1759104

JSON assets: reduce size of task1

f72694f

cvat_data: reduce size of task1

19e7fc9

data.json: reduce size of task1

0422bf0

test tasks: update test

9057719

tesk integrite: fix regexp

53f4b8c

JSON assets:reduce size of task3

9e61bd6

cvat_data:reduce size of task3

0c869c2

data.json:reduce size of task3

c3d028f

JSON assets: reduce size of task4

7b2e2fc

cvat_data: reduce size of task4

d71c5af

data.json: reduce size of task4

254eaab

sizov-kirill changed the title ~~[WIP] Improve REST API testing system~~ Improve REST API testing system Mar 5, 2022

Add cvat_data in unpacked form

8431e21

nmanovic reviewed Mar 9, 2022

View reviewed changes

Use SQL restore script instead of sh

0b6a464

kirill.sizov added 3 commits March 14, 2022 11:04

Return to zip format for cvat_data

9966e45

Update README

cc95cdf

Update README

c7a38f3

kirill.sizov added 3 commits March 14, 2022 15:07

Merge develop into the sk/improve-testing-system

e2ae5db

Resolve conflicts

a906b75

Fix conflicts

f930798

nmanovic reviewed Mar 14, 2022

View reviewed changes

Fix README

5626446

kirill.sizov added 2 commits March 15, 2022 15:08

Update README

8bdcf99

Update README

55cf081

nmanovic approved these changes Mar 16, 2022

View reviewed changes

nmanovic merged commit 6dd73b0 into develop Mar 16, 2022

nmanovic deleted the sk/improve-testing-system branch March 16, 2022 08:49

mikhail-treskin pushed a commit to retailnext/cvat that referenced this pull request Jun 22, 2022

Improve REST API testing system (cvat-ai#4403)

817fcae

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Improve REST API testing system #4403

Improve REST API testing system #4403

sizov-kirill commented Mar 1, 2022 •

edited by nmanovic

Loading

nmanovic Mar 9, 2022

sizov-kirill Mar 10, 2022

nmanovic Mar 9, 2022 •

edited

Loading

sizov-kirill Mar 10, 2022

nmanovic commented Mar 9, 2022

nmanovic commented Mar 9, 2022

sizov-kirill commented Mar 10, 2022 •

edited

Loading

sizov-kirill commented Mar 10, 2022 •

edited

Loading

nmanovic commented Mar 10, 2022

sizov-kirill commented Mar 14, 2022

nmanovic commented Mar 14, 2022

sizov-kirill commented Mar 14, 2022

nmanovic Mar 14, 2022

sizov-kirill Mar 15, 2022

nmanovic Mar 14, 2022

sizov-kirill Mar 15, 2022

nmanovic Mar 14, 2022

sizov-kirill Mar 15, 2022

nmanovic commented Mar 14, 2022

nmanovic commented Mar 14, 2022

sizov-kirill commented Mar 15, 2022

		@@ -0,0 +1,5 @@
		restore="SELECT pg_terminate_backend(pg_stat_activity.pid) FROM pg_stat_activity WHERE pg_stat_activity.datname = 'cvat' AND pid <> pg_backend_pid();\

Improve REST API testing system #4403

Improve REST API testing system #4403

Conversation

sizov-kirill commented Mar 1, 2022 • edited by nmanovic Loading

Motivation and context

How has this been tested?

Checklist

License

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nmanovic Mar 9, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nmanovic commented Mar 9, 2022

nmanovic commented Mar 9, 2022

sizov-kirill commented Mar 10, 2022 • edited Loading

sizov-kirill commented Mar 10, 2022 • edited Loading

nmanovic commented Mar 10, 2022

sizov-kirill commented Mar 14, 2022

nmanovic commented Mar 14, 2022

sizov-kirill commented Mar 14, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nmanovic commented Mar 14, 2022

nmanovic commented Mar 14, 2022

sizov-kirill commented Mar 15, 2022

sizov-kirill commented Mar 1, 2022 •

edited by nmanovic

Loading

nmanovic Mar 9, 2022 •

edited

Loading

sizov-kirill commented Mar 10, 2022 •

edited

Loading

sizov-kirill commented Mar 10, 2022 •

edited

Loading