Manifest #2763

Marishka17 · 2021-02-02T17:35:59Z

Motivation and context

Reducing task creation time for some case
Unified way of working with different data types with prepared meta
Helps to work with remote sources in the future:
- validity of the received data
- reducing of traffic between data storage and server

How has this been tested?

manually, tests

Checklist

I submit my changes into the develop branch
I have added description of my changes into CHANGELOG file
I have updated the documentation accordingly
I have added tests to cover my changes
~~- [ ] I have linked related issues (read github docs)~~
- [ ] I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

License

I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
I have updated the license header for each file (see an example below)

# Copyright (C) 2021 Intel Corporation
#
# SPDX-License-Identifier: MIT

utils/prepare_manifest_file/prepare.py

cvat/apps/engine/media_extractors.py

cvat/apps/engine/migrations/0037_auto_20210127_1354.py

cvat/apps/engine/cache.py

zhiltsov-max · 2021-02-05T21:40:56Z

cvat/apps/engine/media_extractors.py

@@ -25,6 +25,8 @@
 ImageFile.LOAD_TRUNCATED_IMAGES = True

 from cvat.apps.engine.mime_types import mimetypes
+from utils.dataset_manifest import VManifestManager, IManifestManager
+from utils.dataset_manifest.core import WorkWithVideo


I suggest renaming this class to something more descriptive.

@Marishka17 , a class name shouldn't start with a verb. It should describe a type of entity. Could you please rename?

cvat/apps/engine/media_extractors.py

cvat/apps/engine/task.py

utils/dataset_manifest/core.py

coveralls · 2021-02-20T13:29:00Z

Coverage decreased (-0.4%) to 74.854% when pulling 71d1b9a on mk/manifest into 4f7b1f9 on develop.

azhavoro · 2021-02-24T09:25:50Z

@dvkruchinin Could you please take a look at pylint check? It was failed for a reason.

dvkruchinin · 2021-02-24T10:00:07Z

Could you please take a look at pylint check? It was failed for a reason.

Sure. I`ll take a look.

dvkruchinin · 2021-02-24T11:09:13Z

@azhavoro I have prepared a PR #2858 to solve this error.

utils/dataset_manifest/README.md

nmanovic · 2021-03-11T11:47:20Z

utils/dataset_manifest/__init__.py

+# Copyright (C) 2021 Intel Corporation
+#
+# SPDX-License-Identifier: MIT
+from .core import prepare_meta, VideoManifestManager, ImageManifestManager


I will vote for renaming prepare_meta. I don't like the module name. There are multiple variants like dataset_manifest, meta_info, meta_data, etc.

Also it makes sense to move prepare_meta, VideoManifestManager, ImageManifestManager into the module and use dataset_manifest inside core.

I'm not sure that I understand correctly.

prepare_meta - It is a function that responsible for preparing meta information, which later is used to create a manifest. Why it should be dataset_manifest, meta_info ... ?

utils/dataset_manifest/utils.py

utils/dataset_manifest/create.py

nmanovic · 2021-03-18T07:27:40Z

@Marishka17 , comments about the command line:

Let's make dataset_directory positional arugment as an option --output-dir with the current directory as its default value.
Don't analyze the whole video to print a warning about force. Need to adjust the strategy.
Need a progress bar. For long videos users will not understand what is going on. Let's do that in the PR. Let me know if it takes too much time to implement.

zhiltsov-max · 2021-03-18T09:23:02Z

tqdm library can help with progress bars.

zhiltsov-max · 2021-03-19T12:56:01Z

utils/dataset_manifest/create.py

+            try:
+                meta_info = prepare_meta(data_type='video', media_file=source, force=args.force)
+            except AssertionError as ex:
+                if str(ex) == 'Too few keyframes':
                    msg = 'NOTE: prepared manifest file contains too few key frames for smooth decoding.\n' \
                        'Use --force flag if you still want to prepare a manifest file.'
                    print(msg)
                    sys.exit(0)


Maybe it should exit with 2 or something like this?

nmanovic · 2021-03-18T07:36:55Z

utils/dataset_manifest/README.md

+{"type":"video"}
+{"properties":{"name":"video.mp4","resolution":[1280,720],"length":778}}
+{"number":0,"pts":0,"checksum":"17bb40d76887b56fe8213c6fded3d540"}
+{"number":135,"pts":486000,"checksum":"9da9b4d42c1206d71bf17a7070a05847"}


@Marishka17 , Could you please describe how to interpret these data? Just imagine that somebody wants to write code to use these data. Give the user enough information and a couple of sentences. I don't think that you need to describe every parameter, but pts and checksum we should.

Will checksum depends on environment (codec, OS) for video files?

nmanovic · 2021-03-18T08:04:03Z

cvat/apps/engine/media_extractors.py

@@ -25,6 +25,8 @@
 ImageFile.LOAD_TRUNCATED_IMAGES = True

 from cvat.apps.engine.mime_types import mimetypes
+from utils.dataset_manifest import VManifestManager, IManifestManager
+from utils.dataset_manifest.core import WorkWithVideo


@Marishka17 , a class name shouldn't start with a verb. It should describe a type of entity. Could you please rename?

nmanovic · 2021-03-18T08:13:56Z

utils/dataset_manifest/core.py

+
+            for packet in container.demux(video_stream):
+                for frame in packet.decode():
+                    assert frame.pict_type.name == 'I', 'First frame is not key frame'


@Marishka17 , Video is a user input. Please raise an exception instead of assert. Let's treat it as a bad input data (HTTP 400).

nmanovic · 2021-03-18T08:15:43Z

utils/dataset_manifest/core.py

+                    )
+                return frame.width, frame.height
+
+class AnalyzeVideo(WorkWithVideo):


@Marishka17 , What is a reason to split WorkWithVideo and AnalyzeVideo?

Could you please rename class names? Don't start them with a verb. For example, VideoStreamReader or something like that.

nmanovic · 2021-03-18T08:17:00Z

utils/dataset_manifest/core.py

+
+                    frame_pts, frame_dts = frame.pts, frame.dts
+
+class PrepareImageInfo:


Could you please rename? Also probably it is a part of a bigger class which works with images.

nmanovic · 2021-03-18T08:17:37Z

utils/dataset_manifest/core.py

+    def content(self):
+        return self._content
+
+class PrepareVideoInfo(WorkWithVideo):


@Marishka17 , I believe it is a part of VideoStreamReader.

nmanovic · 2021-03-18T08:18:20Z

utils/dataset_manifest/core.py

+        with closing(av.open(self.source_path, mode='r')) as container:
+            self.width, self.height = self._get_frame_size(container)
+
+    def get_task_size(self):


Suggested change

def get_task_size(self):

def get_size(self):

nmanovic · 2021-03-18T10:42:39Z

utils/dataset_manifest/core.py

+    meta_info.create()
+    return meta_info
+
+def prepare_meta(data_type, **kwargs):


@Marishka17 , can it be just a method of ImageManifestManager and VideoManifestManager? What do you think?

nmanovic · 2021-03-18T10:43:26Z

utils/dataset_manifest/core.py

+    def partial_update(self, number, properties):
+        pass
+
+#TODO:


@Marishka17, please leave an appropriate comment here or remove TODO.

…manifest

utils/dataset_manifest/README.md

Marishka17 added 4 commits February 2, 2021 14:51

Added support for manifest file

5335bfe

Added data migration

417e82f

Updated tests

eca3465

Changed script for manually preparing

23792f1

Marishka17 requested review from nmanovic and azhavoro February 2, 2021 17:35

nmanovic reviewed Feb 2, 2021

View reviewed changes

utils/prepare_manifest_file/prepare.py Outdated Show resolved Hide resolved

nmanovic reviewed Feb 2, 2021

View reviewed changes

utils/prepare_manifest_file/prepare.py Outdated Show resolved Hide resolved

nmanovic reviewed Feb 2, 2021

View reviewed changes

cvat/apps/engine/media_extractors.py Outdated Show resolved Hide resolved

nmanovic reviewed Feb 2, 2021

View reviewed changes

cvat/apps/engine/media_extractors.py Outdated Show resolved Hide resolved

nmanovic reviewed Feb 2, 2021

View reviewed changes

cvat/apps/engine/migrations/0037_auto_20210127_1354.py Outdated Show resolved Hide resolved

Marishka17 added 3 commits February 5, 2021 10:38

Fixs

971ebfe

Fixed paths

d8afb6e

some fix & licence headers

6ead5c7

Marishka17 requested a review from zhiltsov-max February 5, 2021 16:17

zhiltsov-max reviewed Feb 5, 2021

View reviewed changes

Marishka17 mentioned this pull request Feb 15, 2021

Support S3 and AWS EFS in Create task #863

Closed

Marishka17 added 5 commits February 17, 2021 18:41

Fixes

2cd2972

Fix stop_frame saving

79d7a36

Merge branch 'upstream/develop' into mk/manifest

64f9ec9

Update migration

97c1746

Fix codacy

cbe1066

Marishka17 changed the title ~~[WIP] Manifest~~ Manifest Feb 24, 2021

Marishka17 added 3 commits February 24, 2021 19:23

Bandit issue & json instead marshal

01c940d

Merge branch 'develop' into mk/manifest

56ea59d

f

7081a96

nmanovic reviewed Mar 11, 2021

View reviewed changes

utils/dataset_manifest/README.md Outdated Show resolved Hide resolved

nmanovic reviewed Mar 11, 2021

View reviewed changes

utils/dataset_manifest/utils.py Show resolved Hide resolved

nmanovic reviewed Mar 11, 2021

View reviewed changes

utils/dataset_manifest/utils.py Show resolved Hide resolved

nmanovic reviewed Mar 11, 2021

View reviewed changes

utils/dataset_manifest/create.py Outdated Show resolved Hide resolved

nmanovic reviewed Mar 11, 2021

View reviewed changes

utils/dataset_manifest/create.py Outdated Show resolved Hide resolved

nmanovic reviewed Mar 11, 2021

View reviewed changes

utils/dataset_manifest/create.py Outdated Show resolved Hide resolved

nmanovic reviewed Mar 11, 2021

View reviewed changes

utils/dataset_manifest/create.py Outdated Show resolved Hide resolved

Marishka17 added 3 commits March 16, 2021 12:50

Refactored script to manually prepare manifest

65e4b12

Update documentation

2d4f456

Merge branch 'upstream/develop' into mk/manifest

3afa428

Marishka17 dismissed zhiltsov-max’s stale review via 3afa428 March 17, 2021 14:50

Fix some comments

38b0d0f

zhiltsov-max reviewed Mar 19, 2021

View reviewed changes

Marishka17 added 2 commits March 22, 2021 12:29

Merge branch 'upstream/develop' into mk/manifest

83ca74d

Fix

c7bbd47

Marishka17 mentioned this pull request Mar 22, 2021

Manifest optimization #2993

Closed

2 tasks

nmanovic reviewed Mar 22, 2021

View reviewed changes

Marishka17 and others added 5 commits March 23, 2021 13:48

One more fix

9c3a81d

Merge branch 'develop' into mk/manifest

7e9389b

Update README

d16022a

Revert prettier changes

3e58fa2

Merge branch 'mk/manifest' of https://github.com/opencv/cvat into mk/…

c0b8a53

…manifest

nmanovic reviewed Mar 24, 2021

View reviewed changes

utils/dataset_manifest/README.md Outdated Show resolved Hide resolved

Update utils/dataset_manifest/README.md

71d1b9a

nmanovic approved these changes Mar 24, 2021

View reviewed changes

nmanovic merged commit 6c38ad0 into develop Mar 24, 2021

nmanovic deleted the mk/manifest branch March 24, 2021 10:47

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Manifest #2763

Manifest #2763

Marishka17 commented Feb 2, 2021 •

edited

zhiltsov-max Feb 5, 2021

nmanovic Mar 18, 2021

coveralls commented Feb 20, 2021 •

edited

azhavoro commented Feb 24, 2021

dvkruchinin commented Feb 24, 2021

dvkruchinin commented Feb 24, 2021

nmanovic Mar 11, 2021

nmanovic Mar 11, 2021

Marishka17 Mar 17, 2021

nmanovic commented Mar 18, 2021

zhiltsov-max commented Mar 18, 2021

zhiltsov-max Mar 19, 2021

nmanovic Mar 18, 2021

nmanovic Mar 18, 2021

nmanovic Mar 18, 2021

nmanovic Mar 18, 2021

nmanovic Mar 18, 2021

nmanovic Mar 18, 2021

nmanovic Mar 18, 2021

nmanovic Mar 18, 2021

nmanovic Mar 18, 2021


		frame_pts, frame_dts = frame.pts, frame.dts

		class PrepareImageInfo:

Manifest #2763

Manifest #2763

Conversation

Marishka17 commented Feb 2, 2021 • edited

Motivation and context

How has this been tested?

Checklist

License

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Feb 20, 2021 • edited

azhavoro commented Feb 24, 2021

dvkruchinin commented Feb 24, 2021

dvkruchinin commented Feb 24, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nmanovic commented Mar 18, 2021

zhiltsov-max commented Mar 18, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Marishka17 commented Feb 2, 2021 •

edited

coveralls commented Feb 20, 2021 •

edited