Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sync resumption for SoUD syncs and cleanup handling #8183

Merged
merged 26 commits into from
Oct 22, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
eddf5a6
Integrate Morango sync resumption, keeping SoUD sync session alive be…
bjester Jun 30, 2021
bebaa57
Fix cert filter handling, alter sync resume logic
bjester Aug 17, 2021
c35ba79
Server-side cleanup of SoUD syncs on disconnect of SoUD instance
bjester Sep 9, 2021
da3845a
Test fixes: correct facility parameter
bjester Sep 9, 2021
a5568de
Update morango and update call to its cleanup command
bjester Sep 9, 2021
1014e11
Fixes from testing
bjester Sep 9, 2021
f79c62d
Simplify client disconnect cleanup, fix bugs
bjester Sep 9, 2021
449c44a
Integration test
bjester Sep 9, 2021
cb46cf1
Merge branch 'release-v0.15.x' into morango-async
bjester Sep 9, 2021
8c32b84
Aggressively cleanup syncs when errored out, simplify func arg
bjester Sep 9, 2021
47d338c
Fix missing pk
bjester Sep 9, 2021
4e55fab
Quicker resync on resume error, and don't re-raise
bjester Sep 10, 2021
cdbb3ba
Contextualize Zeroconf handling into individual classes, and avoid re…
bjester Sep 21, 2021
2e676a5
Integrate Zeroconf broadcast directly into process plugin, use magicb…
bjester Sep 28, 2021
3fc3139
Update SoUD cleanup to do as much as possible in a Job
bjester Sep 28, 2021
7d0f85c
Fixed merge conflicts
bjester Sep 28, 2021
febd8b9
Add comment regarding service browser
bjester Sep 29, 2021
94c233a
Update zeroconf and fix tests
bjester Oct 1, 2021
fe12a7f
Fix non-SoUD attempts to stop syncs
bjester Oct 6, 2021
b5e41ca
Use task decorator
bjester Oct 6, 2021
1815d83
Update how user arg influences sync filter logic
bjester Oct 12, 2021
89a687a
Establish Zeroconf listener after registration
bjester Oct 13, 2021
ef23aa6
Fix renamed command, fix missing ttl on renew, fix repeated zeroconf …
bjester Oct 13, 2021
867b120
Fix sync queuing issue, enhance logging, stop using deprecated warn
bjester Oct 14, 2021
91831ee
Update morango, add more logging, and try to cancel SoUD jobs
bjester Oct 21, 2021
87ceedf
Fix test assertions after removing kwarg format
bjester Oct 21, 2021
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
3 changes: 3 additions & 0 deletions kolibri/core/auth/constants/morango_sync.py
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
from __future__ import unicode_literals

from kolibri.utils import conf


PROFILE_FACILITY_DATA = "facilitydata"
DATA_PORTAL_SYNCING_BASE_URL = conf.OPTIONS["Urls"]["DATA_PORTAL_SYNCING_BASE_URL"]


class ScopeDefinitions(object):
Expand Down
57 changes: 57 additions & 0 deletions kolibri/core/auth/management/commands/resumesync.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
from kolibri.core.auth.constants.morango_sync import DATA_PORTAL_SYNCING_BASE_URL
from kolibri.core.auth.management.utils import get_network_connection
from kolibri.core.auth.management.utils import MorangoSyncCommand


class Command(MorangoSyncCommand):
help = "Allow the syncing of facility data with Kolibri Data Portal or another Kolibri device."

def add_arguments(self, parser):
parser.add_argument(
"--id", type=str, help="ID of an incomplete session to resume sync"
)
parser.add_argument(
"--baseurl", type=str, default=DATA_PORTAL_SYNCING_BASE_URL, dest="baseurl"
)
parser.add_argument("--noninteractive", action="store_true")
parser.add_argument(
"--chunk-size",
type=int,
default=500,
help="Chunk size of records to send/retrieve per request",
)
parser.add_argument(
"--no-push", action="store_true", help="Do not push data to the server"
)
parser.add_argument(
"--no-pull", action="store_true", help="Do not pull data from the server"
)
parser.add_argument(
"--no-provision",
action="store_true",
help="do not create a facility and temporary superuser",
)
parser.add_argument(
"--user",
type=str,
help="for single-user syncing, the user ID of the account to be synced",
)
parser.add_argument(
"--keep-alive",
action="store_true",
help="do not close the sync session",
)

def handle_async(self, *args, **options):
(baseurl, sync_session_id, chunk_size,) = (
options["baseurl"],
options["id"],
options["chunk_size"],
)

# try to connect to server
network_connection = get_network_connection(baseurl)
sync_session_client = network_connection.resume_sync_session(
sync_session_id, chunk_size=chunk_size
)
self._sync(sync_session_client, **options)