Az/import export tasks #3056
Merged
Az/import export tasks #3056
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
azhavoro
requested review from
bsekachev,
nmanovic and
zhiltsov-max
as code owners
zhiltsov-max
reviewed
Apr 2, 2021
zhiltsov-max
reviewed
Apr 2, 2021
zhiltsov-max
reviewed
Apr 2, 2021
bsekachev
reviewed
Apr 5, 2021
bsekachev
reviewed
Apr 5, 2021
|
Tried this patch also. Some found issues:
|
|
@azhavoro , can we avoid limits 500 images, if we import a task? Does it handle properly? |
|
@azhavoro , when are we going to support the same functionality for projects? |
nmanovic
reviewed
Apr 12, 2021
cvat-ui/src/actions/tasks-actions.ts
Outdated
| dispatch(exportTaskFailed(taskInstance.id, error)); | ||
| } | ||
|
|
||
| dispatch(exportTaskSuccess(taskInstance.id)); |
There was a problem hiding this comment.
Probably you can put the line under try. It looks more naturally.
nmanovic
reviewed
Apr 12, 2021
utils/cli/core/definition.py
Outdated
| help='output file' | ||
| ) | ||
| export_task_parser.add_argument( |
There was a problem hiding this comment.
I don't think we need the argument. Let's have something by default.
nmanovic
reviewed
Apr 12, 2021
utils/cli/core/definition.py
Outdated
| ) | ||
| import_task_parser.add_argument( | ||
| '--import_verification_period', |
There was a problem hiding this comment.
I don't think we need the argument. Let's have something by default.
nmanovic
reviewed
Apr 12, 2021
utils/cli/core/core.py
Outdated
| """ Export and download a whole task """ | ||
| url = self.api.tasks_id(task_id) | ||
| export_url = url + '?action=export' | ||
| response = self.session.get(export_url) |
There was a problem hiding this comment.
Looks like exactly the lines are in the beginning of while loop. Move sleep at the end of while and you can remove these 2 lines.
nmanovic
reviewed
Apr 12, 2021
nmanovic
reviewed
Apr 12, 2021
cvat/apps/engine/task.py
Outdated
| from .log import slogger | ||
| from cvat.apps.engine import models | ||
| from cvat.apps.engine.log import slogger | ||
| from cvat.apps.engine.media_extractors import (MEDIA_TYPES, Mpeg4ChunkWriter, |
There was a problem hiding this comment.
Please revert this changes? The indent should be 4 spaces.
nmanovic
reviewed
Apr 12, 2021
| @@ -284,6 +284,7 @@ Go to the [Django administration panel](http://localhost:8080/admin). There you | |||
| 1. Assigned to — is used to assign a task to a person. Start typing an assignee’s name and/or | |||
| choose the right person out of the dropdown list. | |||
| - `Jobs` — is a list of all jobs for a particular task. Here you can find the next data: | |||
There was a problem hiding this comment.
I think you want to revert the file. Otherwise it will affect documentation PR.
nmanovic
reviewed
Apr 12, 2021
| @@ -536,14 +536,13 @@ archive.zip/ | |||
| - `score` (number) - the annotation `score` field | |||
| - arbitrary attributes - will be stored in the `attributes` annotation section | |||
|
|
|||
|
|
|||
| *Note*: there is also a [support for COCO keypoints over Datumaro](https://github.com/openvinotoolkit/cvat/issues/2910#issuecomment-726077582) | |||
There was a problem hiding this comment.
OK. Probably it is a good idea to revert all files which you don't want to change.
nmanovic
reviewed
Apr 12, 2021
cvat-core/src/server-proxy.js
Outdated
| async function importTask(file) { | ||
| const { backendAPI } = config; | ||
|
|
||
| let annotationData = new FormData(); |
There was a problem hiding this comment.
What is the reason behind annotationData name? Should it be payload or something like that?
Marishka17
reviewed
May 12, 2021
utils/cli/core/core.py
Outdated
| url = self.api.tasks + '?action=import' | ||
| response = self.session.post( | ||
| url, | ||
| files={'task_file': open(filename, 'rb')} |
There was a problem hiding this comment.
Maybe add a handler if such file not exists?
There was a problem hiding this comment.
little bit improved, in this case error message will be like this, I think it's enough for user
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/343.zip'
|
@azhavoro , I have found an issue. Multiple times to import the same video task and got the following error a couple of times: |
|
@azhavoro , also the list of tasks isn't updated. Need to refresh the whole site or go to |
|
@azhavoro , I see great process with the layout of the exported archive. Could we do one more step forward?
Can it be implemented from your point of view? Will we loose any information? |
|
Another exception when try to export a task: |
|
@azhavoro , also I see "index.json" file which should be created on the fly. |
|
Now we have tasks.json {
"name": "tracking video",
"bug_tracker": "",
"status": "annotation",
"labels": [
{
"id": 12,
"name": "person",
"color": "#c06060",
"attributes": []
}
],
"subset": "",
"version": "1.0",
"data": {
"chunk_size": 72,
"image_quality": 70,
"start_frame": 0,
"stop_frame": 794,
"storage_method": "file_system",
"storage": "local",
"chunk_type": "imageset"
},
"jobs": [
{
"start_frame": 0,
"stop_frame": 794,
"id": 6,
"status": "annotation"
}
]
}Can we have something like json below? |
|
For annotations.json before: {
"16": {
"version": 3,
"tags": [],
"shapes": [
{
"type": "rectangle",
"occluded": false,
"z_order": 0,
"points": [
971.5869140625,
462.3623046875,
1020.4619140625,
582.5683822631836
],
"frame": 0,
"label_id": 22,
"group": 0,
"source": "manual",
"attributes": []
},
{
"type": "rectangle",
"occluded": false,
"z_order": 0,
"points": [
1190.86328125,
590.4931640625,
1256.9105758666992,
715.9830322265625
],
"frame": 0,
"label_id": 22,
"group": 0,
"source": "manual",
"attributes": []
},
{
"type": "rectangle",
"occluded": false,
"z_order": 0,
"points": [
527.7490234375,
516.5205078125,
717.9652404785156,
721.2671356201172
],
"frame": 0,
"label_id": 22,
"group": 0,
"source": "manual",
"attributes": []
}
],
"tracks": []
}
}And after my suggestions: [
{
"version": 3,
"tags": [],
"shapes": [
{
"type": "rectangle",
"occluded": false,
"z_order": 0,
"points": [
971.5869140625,
462.3623046875,
1020.4619140625,
582.5683822631836
],
"frame": 0,
"label": "person",
"group": 0,
"source": "manual",
"attributes": []
},
{
"type": "rectangle",
"occluded": false,
"z_order": 0,
"points": [
1190.86328125,
590.4931640625,
1256.9105758666992,
715.9830322265625
],
"frame": 0,
"label": "person",
"group": 0,
"source": "manual",
"attributes": []
},
{
"type": "rectangle",
"occluded": false,
"z_order": 0,
"points": [
527.7490234375,
516.5205078125,
717.9652404785156,
721.2671356201172
],
"frame": 0,
"label": "person",
"group": 0,
"source": "manual",
"attributes": []
}
],
"tracks": []
}
] |
|
FYI: Extra |
|
In tasks.json we have "storage" and "storage_method". Do we really need both? |
|
If we export a task from a project, we can't import it back to the project. Is it known issue? We just do not have "Import task" button on the project page. The only way is to import and then move the task to the project (using this way in current implementation we lose all the annotated attributes).
Extra offset of "Create new task" button: Develop: This branch: |
|
I exported a task with a finished job. Imported it and job in annotation status. I suppose the same works for jobs in reviews. Also video task has overlap (5 default). Export, import it and now it has overlap 0. |
|
One more possible problem: Looks like issues aren't dumped together with annotations. |
|
|
nmanovic
reviewed
Jun 8, 2021
| cache_dir = get_export_cache_dir(db_task) | ||
| output_path = osp.join(cache_dir, output_path) | ||
|
|
||
| task_time = timezone.localtime(db_task.updated_date).timestamp() |
There was a problem hiding this comment.
Do we need to adjust time for clearning_job somewhere here? I believe we have multiple races here. Several users can try to backup the same task in parallel. Clearning job can remove our file after all our successful checks.
Races are fine if they don't lead to unexpected results.
nmanovic
approved these changes
Jun 8, 2021
bsekachev
reviewed
Jun 9, 2021
Comment on lines
+193
to
+194
| self._db_task = models.Task.objects.prefetch_related('data__images').select_related('data__video').get(pk=pk) | ||
| self._db_data = self._db_task.data |
There was a problem hiding this comment.
@azhavoro
Probably there is the same problem like in the recent fix.
select_related(data__..) breaks prefetch_related(data__...)
2 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation and context
How has this been tested?
Checklist
developbranchcvat-core, cvat-data and cvat-ui)
License
Feel free to contact the maintainers if that's a concern.