Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix dataset downloading #7864

Merged
merged 43 commits into from
May 28, 2024
Merged
Show file tree
Hide file tree
Changes from 32 commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
d7bb1c1
Make dataset cache lifetime configurable
zhiltsov-max May 2, 2024
7f151b1
Use flock to lock export cache
zhiltsov-max May 3, 2024
f48e807
- use redlock for export cache locking
zhiltsov-max May 7, 2024
4c58bf5
update server deps
zhiltsov-max May 7, 2024
d7fa31d
Merge branch 'develop' into zm/fix-dataset-downloading
zhiltsov-max May 8, 2024
5c22d28
Fix deps after merge
zhiltsov-max May 8, 2024
49baadf
Refactor some code
zhiltsov-max May 8, 2024
de73fe6
fix import
zhiltsov-max May 8, 2024
35ce1f6
Fix linter
zhiltsov-max May 8, 2024
7b671f5
Update changelog
zhiltsov-max May 8, 2024
4a4b5d0
Fix failing cloud storage tests
zhiltsov-max May 9, 2024
05f6dd1
Merge branch 'develop' into zm/fix-dataset-downloading
zhiltsov-max May 9, 2024
2c4be73
Add rough test
zhiltsov-max May 14, 2024
5dc3e56
Refactor code
zhiltsov-max May 14, 2024
4c24e10
Refactor test
zhiltsov-max May 14, 2024
98ad20e
Merge remote-tracking branch 'origin/zm/fix-dataset-downloading' into…
zhiltsov-max May 14, 2024
963f657
Refactor some code
zhiltsov-max May 14, 2024
2bf19db
Update deps
zhiltsov-max May 14, 2024
c0769f0
Add redlock docs link
zhiltsov-max May 14, 2024
087ac42
Add more tests
zhiltsov-max May 14, 2024
0d75d4a
Add more tests
zhiltsov-max May 14, 2024
ea71c7d
Add more tests, update export test
zhiltsov-max May 14, 2024
7085076
Update deps
zhiltsov-max May 16, 2024
f8d142f
Add rq cleanup to tests with multiple iterations
zhiltsov-max May 16, 2024
9479888
Update comment
zhiltsov-max May 16, 2024
da06903
Remove unused imports
zhiltsov-max May 16, 2024
700bf2c
Refactor lock release
zhiltsov-max May 16, 2024
9ccafc5
Rename lock function
zhiltsov-max May 16, 2024
c2b3049
Add api compatibility test
zhiltsov-max May 16, 2024
228e82e
Merge branch 'develop' into zm/fix-dataset-downloading
zhiltsov-max May 16, 2024
4ddc79d
Fix tests
zhiltsov-max May 16, 2024
447b4df
Merge remote-tracking branch 'origin/zm/fix-dataset-downloading' into…
zhiltsov-max May 16, 2024
2a67f2e
Merge branch 'develop' into zm/fix-dataset-downloading
zhiltsov-max May 17, 2024
4471aa9
Restore old job removal behavior
zhiltsov-max May 17, 2024
de2c805
Merge remote-tracking branch 'origin/zm/fix-dataset-downloading' into…
zhiltsov-max May 17, 2024
e76382d
Update downloading test
zhiltsov-max May 17, 2024
2af76a8
Fix comments
zhiltsov-max May 27, 2024
aafc369
Remove unused import
zhiltsov-max May 27, 2024
275a274
Raise an exception on unlocking expired lock
zhiltsov-max May 28, 2024
297d686
Remove unexpected requirements changes
zhiltsov-max May 28, 2024
8d5401c
Merge branch 'develop' into zm/fix-dataset-downloading
zhiltsov-max May 28, 2024
664ed28
Remove extra requirements
zhiltsov-max May 28, 2024
6fc3f63
Merge remote-tracking branch 'origin/zm/fix-dataset-downloading' into…
zhiltsov-max May 28, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
### Fixed

- The 500 / "The result file does not exist in export cache" error
on dataset export request
(<https://github.com/cvat-ai/cvat/pull/7864>)
18 changes: 18 additions & 0 deletions cvat/apps/dataset_manager/apps.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Copyright (C) 2024 CVAT.ai Corporation
#
# SPDX-License-Identifier: MIT

from django.apps import AppConfig


class DatasetManagerConfig(AppConfig):
name = "cvat.apps.dataset_manager"

def ready(self) -> None:
from django.conf import settings

from . import default_settings

for key in dir(default_settings):
if key.isupper() and not hasattr(settings, key):
setattr(settings, key, getattr(default_settings, key))
SpecLad marked this conversation as resolved.
Show resolved Hide resolved
14 changes: 14 additions & 0 deletions cvat/apps/dataset_manager/default_settings.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Copyright (C) 2024 CVAT.ai Corporation
#
# SPDX-License-Identifier: MIT

import os

DATASET_CACHE_TTL = int(os.getenv("CVAT_DATASET_CACHE_TTL", 60 * 60 * 10))
"Base lifetime for cached exported datasets, in seconds"

DATASET_CACHE_LOCK_TIMEOUT = int(os.getenv("CVAT_DATASET_CACHE_LOCK_TIMEOUT", 10))
"Timeout for cache lock acquiring, in seconds"

DATASET_EXPORT_LOCKED_RETRY_INTERVAL = int(os.getenv("CVAT_DATASET_EXPORT_LOCKED_RETRY_INTERVAL", 60))
"Retry interval for cases the export cache lock was unavailable, in seconds"