Extend COCO format to skeletons #4821

yasakova-anastasia · 2022-08-24T07:38:07Z

Motivation and context

Added COCO Keypoints 1.0 for skeleton import/export
Fixed an issue with moving tasks between projects

How has this been tested?

Checklist

I submit my changes into the develop branch
I have added a description of my changes into CHANGELOG file
I have updated the documentation accordingly
I have added tests to cover my changes
I have linked related issues (read github docs)
I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

License

I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.

zhiltsov-max · 2022-08-26T17:02:23Z

Tried to export task inside project with 3 skeletons in COCO and then import back. It seems that visibility properties of points are not preserved - neither occluded, nor outside. After import I've got all points visible. The exported files contain this info.
Failed to export this task in CVAT format, backups for the task and project also failed

# task export
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/dataset_manager/views.py", line 114, in export_task_as_dataset
    return export(dst_format, task_id=task_id, server_url=server_url, save_images=True)
  File "/home/django/cvat/apps/dataset_manager/views.py", line 81, in export
    export_fn(db_instance.id, temp_file, dst_format,
  File "/home/django/cvat/apps/dataset_manager/task.py", line 750, in export_task
    task.init_from_db()
  File "/home/django/cvat/apps/dataset_manager/task.py", line 622, in init_from_db
    annotation.init_from_db()
  File "/home/django/cvat/apps/dataset_manager/task.py", line 535, in init_from_db
    self._init_shapes_from_db()
  File "/home/django/cvat/apps/dataset_manager/task.py", line 437, in _init_shapes_from_db
    shapes[db_shape.parent].elements.append(db_shape)
KeyError: 45

# task backup
Traceback (most recent call last):
  File "/opt/venv/lib/python3.8/site-packages/rq/worker.py", line 936, in perform_job
    rv = job.perform()
  File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 684, in perform
    self._result = self._execute()
  File "/opt/venv/lib/python3.8/site-packages/rq/job.py", line 690, in _execute
    return self.func(*self.args, **self.kwargs)
  File "/home/django/cvat/apps/engine/backup.py", line 710, in _create_backup
    exporter.export_to(temp_file)
  File "/home/django/cvat/apps/engine/backup.py", line 386, in export_to
    self._export_task(zip_obj=zf, target_dir=target_dir)
  File "/home/django/cvat/apps/engine/backup.py", line 377, in _export_task
    self._write_annotations(zip_obj, target_dir)
  File "/home/django/cvat/apps/engine/backup.py", line 369, in _write_annotations
    annotations = serialize_annotations()
  File "/home/django/cvat/apps/engine/backup.py", line 362, in serialize_annotations
    annotations = dm.task.get_job_data(db_job_id)
  File "/home/django/cvat/apps/profiler.py", line 11, in wrapped
    return f(*args, **kwargs)
  File "/usr/lib/python3.8/contextlib.py", line 75, in inner
    return func(*args, **kwds)
  File "/home/django/cvat/apps/dataset_manager/task.py", line 659, in get_job_data
    annotation.init_from_db()
  File "/home/django/cvat/apps/dataset_manager/task.py", line 535, in init_from_db
    self._init_shapes_from_db()
  File "/home/django/cvat/apps/dataset_manager/task.py", line 437, in _init_shapes_from_db
    shapes[db_shape.parent].elements.append(db_shape)
KeyError: 45

It seems that I can't import skeletons from a dataset. When I try to import the exported task data as a dataset for an empty project, I get this error:

Error: Request failed with status code 500. "cvat.apps.dataset_manager.bindings.CvatImportError: Image img3: can't import annotation #0 (points): Label '1' is not registered for this task\n".

Failed to move task to another project with the same skeleton count, but the skeletons are different (including different point count). The error is:

.3:60250] Traceback (most recent call last):
[Fri Aug 26 17:00:40.001177 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
[Fri Aug 26 17:00:40.001178 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     response = get_response(request)
[Fri Aug 26 17:00:40.001179 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
[Fri Aug 26 17:00:40.001181 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     response = wrapped_callback(request, *callback_args, **callback_kwargs)
[Fri Aug 26 17:00:40.001182 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
[Fri Aug 26 17:00:40.001183 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return view_func(*args, **kwargs)
[Fri Aug 26 17:00:40.001183 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
[Fri Aug 26 17:00:40.001184 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return self.dispatch(request, *args, **kwargs)
[Fri Aug 26 17:00:40.001185 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
[Fri Aug 26 17:00:40.001186 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     response = self.handle_exception(exc)
[Fri Aug 26 17:00:40.001187 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
[Fri Aug 26 17:00:40.001188 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     self.raise_uncaught_exception(exc)
[Fri Aug 26 17:00:40.001189 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
[Fri Aug 26 17:00:40.001190 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     raise exc
[Fri Aug 26 17:00:40.001191 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
[Fri Aug 26 17:00:40.001192 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     response = handler(request, *args, **kwargs)
[Fri Aug 26 17:00:40.001193 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/drf_spectacular/drainage.py", line 156, in wrapped_method
[Fri Aug 26 17:00:40.001194 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return method(self, request, *args, **kwargs)
[Fri Aug 26 17:00:40.001195 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/mixins.py", line 82, in partial_update
[Fri Aug 26 17:00:40.001196 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return self.update(request, *args, **kwargs)
[Fri Aug 26 17:00:40.001197 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/drf_spectacular/drainage.py", line 156, in wrapped_method
[Fri Aug 26 17:00:40.001198 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return method(self, request, *args, **kwargs)
[Fri Aug 26 17:00:40.001199 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/mixins.py", line 68, in update
[Fri Aug 26 17:00:40.001200 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     self.perform_update(serializer)
[Fri Aug 26 17:00:40.001200 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/home/django/cvat/apps/engine/views.py", line 773, in perform_update
[Fri Aug 26 17:00:40.001201 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     updated_instance = serializer.save()
[Fri Aug 26 17:00:40.001202 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/rest_framework/serializers.py", line 200, in save
[Fri Aug 26 17:00:40.001203 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     self.instance = self.update(self.instance, validated_data)
[Fri Aug 26 17:00:40.001204 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/home/django/cvat/apps/engine/serializers.py", line 632, in update
[Fri Aug 26 17:00:40.001205 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     model.objects.filter(job__segment__task=instance, label=old_label).update(
[Fri Aug 26 17:00:40.001206 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/models/query.py", line 783, in update
[Fri Aug 26 17:00:40.001207 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     rows = query.get_compiler(self.db).execute_sql(CURSOR)
[Fri Aug 26 17:00:40.001208 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1559, in execute_sql
[Fri Aug 26 17:00:40.001209 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     cursor = super().execute_sql(result_type)
[Fri Aug 26 17:00:40.001209 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/models/sql/compiler.py", line 1175, in execute_sql
[Fri Aug 26 17:00:40.001210 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     cursor.execute(sql, params)
[Fri Aug 26 17:00:40.001211 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 98, in execute
[Fri Aug 26 17:00:40.001213 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return super().execute(sql, params)
[Fri Aug 26 17:00:40.001214 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 66, in execute
[Fri Aug 26 17:00:40.001215 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
[Fri Aug 26 17:00:40.001216 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 75, in _execute_with_wrappers
[Fri Aug 26 17:00:40.001217 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return executor(sql, params, many, context)
[Fri Aug 26 17:00:40.001218 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
[Fri Aug 26 17:00:40.001218 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return self.cursor.execute(sql, params)
[Fri Aug 26 17:00:40.001219 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/utils.py", line 90, in __exit__
[Fri Aug 26 17:00:40.001220 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     raise dj_exc_value.with_traceback(traceback) from exc_value
[Fri Aug 26 17:00:40.001221 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]   File "/opt/venv/lib/python3.8/site-packages/django/db/backends/utils.py", line 84, in _execute
[Fri Aug 26 17:00:40.001222 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250]     return self.cursor.execute(sql, params)
[Fri Aug 26 17:00:40.001223 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250] django.db.utils.IntegrityError: null value in column "label_id" violates not-null constraint
[Fri Aug 26 17:00:40.001224 2022] [wsgi:error] [pid 116:tid 140703314855680] [remote 172.22.0.3:60250] DETAIL:  Failing row contains (51, 0, 0, points, f, 0, 469.69,307.0, 7, null, manual, 0, 42, f).

Can't open a job in the task after failed move

yasakova-anastasia · 2022-08-29T12:14:52Z

@zhiltsov-max, thank you for the review.

Tried to export task inside project with 3 skeletons in COCO and then import back. It seems that visibility properties of points are not preserved - neither occluded, nor outside. After import I've got all points visible. The exported files contain this info.

Fixed.

Failed to export this task in CVAT format, backups for the task and project also failed

I cannot reproduce this problem.

It seems that I can't import skeletons from a dataset. When I try to import the exported task data as a dataset for an empty project, I get this error:
Error: Request failed with status code 500. "cvat.apps.dataset_manager.bindings.CvatImportError: Image img3: can't import annotation #0 (points): Label '1' is not registered for this task\n".

I'm aware of this issue. This is not related to COCO format. I will fix it in another PR.

Failed to move task to another project with the same skeleton count, but the skeletons are different (including different point count).

Fixed.

Can't open a job in the task after failed move

Fixed.

sizov-kirill · 2022-08-29T16:11:32Z

Question: Am I understand correctly that for moving task from one project to another I should to setup skeleton label twice: ones for source project and once for target project and these skeletons must be identical?

Error: I put skeleton on an image and marked one point as outside, after that bounding box of my skeleton changed, I saved, and after that I tried to export annotations to the COCO Keypoints 1.0 format, but I got an error Error: Request failed with status code 500. "KeyError: 127\n".

yasakova-anastasia · 2022-08-29T16:24:13Z

Question: Am I understand correctly that for moving task from one project to another I should to setup skeleton label twice: ones for source project and once for target project and these skeletons must be identical?

Yes, you're right

sizov-kirill · 2022-08-31T09:13:14Z

Could you please merge develop branch, it has some important changes that necessary for running all tests for this PR?

yasakova-anastasia · 2022-08-31T13:45:16Z

/check

github-actions · 2022-08-31T13:45:30Z

❌ Some checks failed
📄 See logs here

sizov-kirill · 2022-09-01T06:38:11Z

/check

github-actions · 2022-09-01T06:38:26Z

❌ Some checks failed
📄 See logs here

sizov-kirill

LGTM, but need to add documentation for new format.

bsekachev and others added 30 commits June 30, 2022 16:17

Rotation for skeleton shapes

9ecf9f3

Extended CVAT for images for skeletons (import)

f2a10e0

Fixed an issue with labels

7281ff0

Rotation for tracks

719e5e2

Merge branch 'bs/skeletons' of github.com:cvat-ai/cvat into bs/skeletons

798f919

Fixed issue with saving

8d6d710

Some code refactored

f7debf4

Rotatable text

b54428f

Fixed resizing/dragging for rotated shapes

8a951fc

Fixed rotation 0:360

1d23228

Added skeletons to labels for projects

63a3a74

Extended CVAT for video for skeletons (export)

57de335

Merge remote-tracking branch 'origin/bs/skeletons' into bs/skeletons

7c95626

Fixed pasting rotated skeleton

fc1a7a0

Merge branch 'bs/skeletons' of github.com:cvat-ai/cvat into bs/skeletons

44a7001

Fixed an issue with updating labels

25cda10

tmp

9231f4c

Added interpolation for skeletons on the server

c77c88c

Fixed an issue with track export

ff5ca8d

Fixed an issue with track import

53aef79

Merge remote-tracking branch 'origin/bs/skeletons' into bs/skeletons

19fba47

tmp

0a51f50

Merge branch 'bs/skeletons' of github.com:cvat-ai/cvat into bs/skeletons

1d373be

Redesigned rotation approach

564a56c

Reduced number of code, refactoring

0a3ff37

Undo redo for shapes

c036579

Fixed couple of issues with drawing using N

740fa58

Do not show the whole labels list on UI

bd34112

Fixed creating annotation

1607475

Enabled z order

8e4b2c6

yasakova-anastasia added 3 commits August 29, 2022 13:51

Fixed an issue with outside and occluded

fda6f9e

resolved conflicts

702dcf0

Fixed an issue with moving tasks

cda946c

yasakova-anastasia requested a review from sizov-kirill August 29, 2022 12:43

yasakova-anastasia added 3 commits August 30, 2022 12:37

Some fixes

c29ae95

Resolved conflicts

1b7886f

Fixed tests

e8e60bb

cvat-ai deleted a comment from github-actions bot Aug 31, 2022

yasakova-anastasia added 3 commits August 31, 2022 12:23

Merge branch 'develop' into ay/skeletons-to-coco

ad4e650

Removed useless changes

bec3543

Fixed tests

e550c06

cvat-ai deleted a comment from github-actions bot Aug 31, 2022

yasakova-anastasia added 2 commits August 31, 2022 16:43

Added test

d65823f

Fixed unit tests

8dce400

cvat-ai deleted a comment from github-actions bot Aug 31, 2022

sizov-kirill approved these changes Sep 1, 2022

View reviewed changes

nmanovic approved these changes Sep 1, 2022

View reviewed changes

nmanovic merged commit 8b3e247 into develop Sep 1, 2022

nmanovic deleted the ay/skeletons-to-coco branch September 1, 2022 08:51

nmanovic mentioned this pull request Sep 12, 2022

Release v2.2.0 #4936

Merged

yasakova-anastasia mentioned this pull request Oct 21, 2022

Support of (Key-)Point formats in CVAT #2910

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extend COCO format to skeletons #4821

Extend COCO format to skeletons #4821

yasakova-anastasia commented Aug 24, 2022 •

edited

Loading

zhiltsov-max commented Aug 26, 2022 •

edited

Loading

yasakova-anastasia commented Aug 29, 2022

sizov-kirill commented Aug 29, 2022 •

edited

Loading

yasakova-anastasia commented Aug 29, 2022

sizov-kirill commented Aug 31, 2022

yasakova-anastasia commented Aug 31, 2022

github-actions bot commented Aug 31, 2022 •

edited

Loading

sizov-kirill commented Sep 1, 2022

github-actions bot commented Sep 1, 2022 •

edited

Loading

sizov-kirill left a comment

Extend COCO format to skeletons #4821

Extend COCO format to skeletons #4821

Conversation

yasakova-anastasia commented Aug 24, 2022 • edited Loading

Motivation and context

How has this been tested?

Checklist

License

zhiltsov-max commented Aug 26, 2022 • edited Loading

yasakova-anastasia commented Aug 29, 2022

sizov-kirill commented Aug 29, 2022 • edited Loading

yasakova-anastasia commented Aug 29, 2022

sizov-kirill commented Aug 31, 2022

yasakova-anastasia commented Aug 31, 2022

github-actions bot commented Aug 31, 2022 • edited Loading

sizov-kirill commented Sep 1, 2022

github-actions bot commented Sep 1, 2022 • edited Loading

sizov-kirill left a comment

Choose a reason for hiding this comment

yasakova-anastasia commented Aug 24, 2022 •

edited

Loading

zhiltsov-max commented Aug 26, 2022 •

edited

Loading

sizov-kirill commented Aug 29, 2022 •

edited

Loading

github-actions bot commented Aug 31, 2022 •

edited

Loading

github-actions bot commented Sep 1, 2022 •

edited

Loading