Make the importer more stable

[christian-bernet]

No description provided.

[christian-bernet]

Estimate 4 hours

[christian-bernet]

add a clean up process if the state is inconsistence & validate json files!

[interrogator]

As part of this issue, let me know if there is something I can add to the JSON validator, which basically checks that JSON is valid, and that mandatory keys are there, containing values of correct type.

[christian-bernet]

should be better now.
We changed the importer to a simple concept.
@pim can you please build the new images for the importer, so that we have the new code place? many thanks.

[christian-bernet]

containers where updated by pim right now.

[martindusinberre]

@christian-bernet Can you please give us a bit of help on something that happened on Friday afternoon? I was in OurTestTopic and I wanted to see if a functionality worked in checkboxExercise. So I changed one value ("minimum" to "maximum") and pushed a change. The log told me that the import hadn't worked so I concluded that it's not possible to have a "maximum" value. (That's fine--I just wanted to know.) So I then changed the json files back to what they had been and pushed another change.
By my understanding, this should then import fine (all the json values were correct). But instead of importing properly, the whole OurTestTopic crashed and dropped from the Topic Selection page.
According to @witlox , he couldn't resolve the issue by following the documentation you provided for Danny a couple of weeks ago. Given that it was urgent (I had three colleagues needing to look at OurTestTopic over the weekend), Pim effectively made a new folder, one which doesn't automatically update. This is a good work-around but not a great solution because it means that I have to ask Danny or Pim to do a manual import whenever I want to introduce new material.
So the question is: do you happen to know why this happened? And what we can do about it in the future? Is it a bug in the system, so that if the import is rejected once, it continues to be rejected (even if the new change is correct)? Or is this a problem with the logic?
Any comments much appreciated... :)

[christian-bernet]

@martindusinberre
I think the importer was broken and after the steps in the wiki it should run again.
https://github.com/uzh/marugoto/wiki/Infrastructure
This does not happens on a specific situation. It has nothing to do with your changes.

As I see, the import is still broken for the test topic.
I think it would be best, that Pim and or Daniel can have a look of how we can clean it up, so that the importer will run again.

If it's ok for you like that, they should come to me and I can show them.

[martindusinberre]

@christian-bernet That would be perfect, great. Can you please amend this issue to a "bug" and assign also to Danny? Many thanks, Martin
(PS: if you are all working on the importer / testtopic, could you at the same time create a new testtopic environment for Lena to start directly importing her own storyline content?)

[christian-bernet]

For me this Wednesday would work best, to do that.

[christian-bernet]

done with: e433c9c

[christian-bernet]

Let us see if there will be some problem in the next weeks with this simplified version of the importer. I will leave this issue open until we know more.

[martindusinberre]

@interrogator could you give a quick narrative of what you discussed with Christian last Wednesday for this, just for documentation purposes? Then I would close it (possibly as reopenable, depending on how good you think the fixes are).

[interrogator]

I got a set of instructions that aren't much use here:

2000 ./shell
2002 cd /content/
2007 sudo rm .OnlyTest/ -r
2008 cd
2010 rm test.checkpoint
2022 tail -f /var/www/html/log/importer_test.log
in the ./shell (command 2000)
I ran these commands:
1. list-topics
2. set-topic-state false 16464401
and quit with "exit"

And as Christian said, the updated code is diffed here: e433c9c . Soon I bet we'll be importing more often, and will see if it truly is more stable. Anecdotal evidence so far is that the importer has been OK lately.

[martindusinberre]

Yes, I agree--anecdotally it seems to be working better (famous last words!). So I'll close this but mark as reopenable.