Start developing Debate XML import/export #1076
Conversation
This commit starts creating an interface for the import/export and implementation of the Debate XML specification. Here, a side-wide view for importing an XML file is created, with a consumer link for async processing.
tienne-B
changed the title
This commit adds a tournament url for tournament export with a stub page.
|
I'm a bit concerned I'm putting the cart before the horse here, as the schema has not been reviewed or discussed: TabbycatDebate/DebateXML#4. @czlee, @philipbelesky: Mind looking it over (even just cursorily)? I should get back to this in a week or so. |
|
Happy to look over it, although might need to wait a few days. |
|
I've mostly commented over in the DTA repo, but just was curious about the desire to write an importer and whether you were planning to do that before/after/simultaneous with the exporter? It's certainly an interesting idea. Initially I pictured the point of DTA as enabling much more of an archival-storage/ data-vis purpose, but I guess having a Tabbycat importer means that even if the spec doesn't take off more broadly we would have a robust way of mitigating against data loss and/or combining tournaments into a single instance. |
|
An importer would be useful for me as a way to collate various tournaments previously on individual sites to a central site, of which is needed by SUCDI for end-of-year prizes. Having the importer would also be useful as TC already has some data visualisations and analyses (and result views) that would make the archive format more accessible. I am creating the importer simultaneously with the exporter as I think having the importer gives a visible meaning for the exporter (although that isn't the main point); aiding with adoption. |
tienne-B
force-pushed
the
debate-xml
branch
8 times, most recently
from
26887db
to
b632ee9
Compare
This commit adds an exporter for single tournaments into an XML format as specified in dta-spec.
|
So exporting seems to be working well but with N+1 SQL inefficiencies that I'm finding hard to resolve in the In order to expose the debug toolbar to view the SQL queries, I add a typo in another method. |
|
Hmm, I think to avoid |
|
Sorry, I don't understand on the points of "fetching without any reference of current status", nor what would be the threshold between inelegance and being severe. On the first, is it an inability to prefetch a table in various ways (e.g. from |
A new page is added detailing the XML format and how to import/export it to/from Tabbycat. A distinction has been made between it and database backups.
tienne-B
force-pushed
the
debate-xml
branch
4 times, most recently
from
88c173f
to
051c100
Compare
Also separate methods for adding debates, add gender to people, and removing team short name.
This commit starts an importer for Debate XML (The "Agora Project") with support for all that can be exported by Tabbycat. Some considerations are marked in the documentation page about this format.
|
Umm… yeah. Unless it's painfully slow, like, more than a minute or so. But otherwise I think I'd rather display a progress bar and have it more robust to future changes in I'm open to discussion on this, though. |
|
Understood. I'll prepare a patch for performance comparison (after #1180 as it would improve result importation). Having a progress bar would be quite difficult as import does not use consumers. |
|
Hmm, should the import use consumers? Normally this is best practice for anything that takes a nontrivial amount of time? (It's fine if you don't want to do this for a first implementation! Just thinking out loud.) |
Some methods have also been added to DebateResult classes for the change.
|
This PR depends on #1180 for the use of |
To be able to use new DebateResult classes in importing/exporting.
This commit replaces the uses of `.bulk_create()`, instead saving each DB object one-by-one. This is to trigger the `.save()` of the models and the triggers. The results importer has not been touched, and will be modified to make use of DebateResult in another commit.
This commit replaces all the logic to determine the result of a debate, such as the margin/splits, etc. with completing a DebateResult object and saving that. This commit refers to methods created as part of TabbycatDebate#1180.
This is necessary as get_winner() has the same effect.
This commit fixes the DebateResult generation as well as some typos and omissions caused from previous changes.
Also rename uses of 'test score'.
As booleans must be either 'true' or 'false' in XML (note the use of lowercase), rather than the capitalized keywords used in Python.
# Conflicts: # config/requirements_core.txt # tabbycat/importer/urls.py # tabbycat/importer/views.py # tabbycat/options/views.py # tabbycat/results/dbutils.py # tabbycat/results/result.py # tabbycat/results/utils.py # tabbycat/urls.py
Even though email addresses and conflicts would not be typically added to an exported XML, the importer should still use them if they exist.
This commit starts creating an interface for the import/export and implementation of the Debate XML specification. Here, a side-wide view for importing an XML file is created, with a consumer link for asynchronous processing.
More to come