Move state of reports to the filesystem

[hellais]

This implements one of the solutions outlined in #86 to support saving the state of a ooni-backend instance when it is shutdown.

Basically we have two files for an opened report with ID $REPORT_ID they both reside in the $REPORTS_DIR.

1. The file at the path $REPORTS_DIR/$REPORT_ID is the actual partial report, that get's updated every time we need to create a new entry.
2. The file at the path $REPORTS_DIR/$REPORT_ID-metadata.json contains the metadata associated with this report (the metadata sent on open()).

Every time we get a new update request we append to file 1. and touch file 2.

At regular intervals we look inside of the $REPORTS_DIR for all files ending with -metadata.json and look at their mtime with stat. If they have not been modified in a time frame greater than the stale_time we consider them stale and close them.

[coveralls]

Coverage decreased (-0.2%) to 61.438% when pulling 89e2e46 on feature/fs-db into 6bd4a24 on master.

[bassosimone]

Epic! Will review and test this!

[bassosimone]

For symmetry with moving APIs in class scope you would probably want to move also this statement

[bassosimone]

• diff looks good
• testing using MK as input
• testing using MK instrumented not to close the report
• testing using old YAML format
• shutdown and restart of oonib while ooniprobe is running a deck
• check that stale reports are pruned
• some more testing
• merge

---

The test where I shut down and then restarted oonib while ooniprobe was running a deck was quite interesting. I lost some entries, of course, but after oonib came up again, everything continued to work and subsequent entries were appended to the report file.

---

As regards testing using the old YAML format, I checked out v1.3.2 and run the following command (the same I used for testing restart of backend while ooni v1.5.0dev deck was in progress):

./bin/ooniprobe-dev -c http://127.0.0.1:10081 -i deck-it/0.1.0-it-user.deck

What I obtain is this error and of course no report is created:

[!] Failed to connect to reporter backend
Traceback (most recent call last):
  File "/Users/sbasso/src/TheTorProject/ooni-probe/ooni/reporter.py", line 333, in createReport
    response = yield self.agent.request("POST", url,
AttributeError: 'OONIBReporter' object has no attribute 'agent'

@hellais, any hint?

[hellais]

I believe the issue lies in the fact that ooniprobe 1.3.2 doesn't support non httpo collectors (i.e. https or http). I would advise you run it by specifying the collector address in httpo format.

[coveralls]

Coverage increased (+0.6%) to 62.204% when pulling 28caaaa on feature/fs-db into 6bd4a24 on master.

[bassosimone]

I would advise you run it by specifying the collector address in httpo format.

Yes, this strategy is working, thanks!

[bassosimone]

I've done more testing! Going to merge!