MDB_KEYEXIST errors

[neilmayhew]

When I run asgen on my full repo (which has some empty sections) I get exceptions as follows:

object.Exception@source/datacache.d(85): mdb_put (stats)[-30799]: MDB_KEYEXIST: Key/data pair already exists
----------------
0x51bb51 void ag.datacache.DataCache.checkError(int, immutable(char)[])
    source/datacache.d:85
0x51eef9 void ag.datacache.DataCache.addStatistics(immutable(char)[])
    source/datacache.d:569
0x5377bd void ag.reportgenerator.ReportGenerator.saveStatistics(immutable(char)[], immutable(char)[], ag.reportgenerator.ReportGenerator.DataSummary)
    source/report-generator.d:634
0x5395d6 void ag.reportgenerator.ReportGenerator.processFor(immutable(char)[], immutable(char)[], ag.backend.intf.Package[])
    source/report-generator.d:722
0x5225c6 void ag.engine.Engine.run(immutable(char)[])
    source/engine.d:381
0x453af8 _Dmain
    source/app.d:112
0x60026e _D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv
    ../../../../src/libphobos/libdruntime/rt/dmain2.d:411
0x60039e void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())
    ../../../../src/libphobos/libdruntime/rt/dmain2.d:386
0x600778 void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).runAll()
    ../../../../src/libphobos/libdruntime/rt/dmain2.d:411
0x60039e void rt.dmain2._d_run_main(int, char**, extern (C) int function(char[][])*).tryExec(scope void delegate())
    ../../../../src/libphobos/libdruntime/rt/dmain2.d:386
0x600525 _d_run_main
    ../../../../src/libphobos/libdruntime/rt/dmain2.d:419
0x45367a main
    /usr/lib/gcc/x86_64-linux-gnu/5/include/d/__entrypoint.di:62
0x7f5a08a5660f __libc_start_main
    ???:0
0x453588 _start
    ???:0
0xffffffffffffffff ???
    ???:0

This occurs 8 times, and I have 8 suites configured:
{trusty,xenial}{,-{updates,proposed,experimental}}
This may be coincidence, but we do often have the same package appear in multiple suites if it's arch-independent and simply hasn't needed a rebuild recently.

Here's the complete output of the asgen run: asgen-out.txt.zip

If you need access to the entire repository, I can give you anonymous rsync access — PM me for details. It's about 8.5GB.

[neilmayhew]

I did rm -r cache export beforehand, too

[ximion]

Hmm... Those empty repos really are a pain... Also, you seem to have a fast machine ^^
Can you try with the patch I above (committed to master) and check if that fixes it?

[neilmayhew]

Looking through the 'issues' reports, I've found that one of our packages uses the same .desktop file name in different versions of the package within the same same suite. To be specific, we have paratext, paratext-7.4, paratext-7.5 and paratext-7.6 that all contain ./usr/share/applications/paratext.desktop. Maybe this is causing the error.

[neilmayhew]

No exceptions! So that seems to have fixed it. Thanks for your great responsiveness on this and the other issues.

Yes, this is a nice machine. i7 overclocked to 4.7GHz equivalent, 16GB RAM, 500GB SSD used as a bcache to a larger HD.

The 9e886e3 commit didn't slow it down at all, BTW. Real time was the same in both cases, starting without cache.

[ximion]

Neat :-)
The origin of this issue is the way we store statistical information in the database, so it has nothing to do with the extraction itself.
The paratext.desktop thing is an issue though, you should select one of those as default and mark the other ones with X-AppStream-Ignore so they are ignored by the generator.

The plan is do add some code later to remove duplications like this by emitting appropriate issue hints.

[neilmayhew]

These exceptions are still occurring for me, especially when re-running after nothing has changed (presumably because that's a lot faster so there's more parallel processing going on). I looked at what you'd done in 9e886e3 and increased the count to 100. That seems to fix it, and doesn't decrease the running time much, but it seems like a very brute-force approach and I wonder if there needs to be a design change to the way stats keys are generated.

Also, I think this does affect the extraction itself because the exception seems to abort the processing of an entire section.

[ximion]

I wonder if there needs to be a design change to the way stats keys are generated.

Definitely ;-)
It should just be backwards-compatible, which is the issue at time, and the reason why I did that quick fix. I'm still thinking about this.
The processing isn't aborted, but the exporting of data and report-generation is (which looks the same, but really isn't, since the next run won't have to do much besides exporting the data).

I think you run asgen much more often than anyone before, on a much faster machine, that's why your processing finishes too quick ;-)

One possible permanent fix for this is to turn the key-value nature of the statistics sub-database into some log-like behavior, allowing multiple values per key. That just needs to be represented properly in the UI, so the second option, which I prefer at time, is to extend the existing value at the given time with new data.

[ximion]

The latest change in master should fix this problem without hacks now :)

[neilmayhew]

The latest change in master should fix this problem

It does away with the errors, yes, but isn't there still a problem? At datacache.d:586 the assignment to the stats array will overwrite previous stats that have the same key, and of the multiple values that are now being saved with the same key, only the last one will be seen. The multiple values could be for different sections in the same suite, or even for different suites, and this would ignore data from some suites and sections. I think you need to unpack the JSON, merge it with the existing data, and repack it. (Or do the same with an equivalent D data structure.)

BTW, I've figured out the main reason why I was hitting this error. I often re-run the generator when there haven't been any changes to the repo (remember this is a small, 3rd-party repo that doesn't get updates every day) so the generation run is very quick. The whole thing takes under 3 seconds so of course there are a lot of conflicts since I have a total of 128 sections ({trusty,wily,xenial,yakkety}{,-updates,-proposed,-experimental}/{main,restricted,universe,multiverse}/{amd64,i386}). Even if I did only one suite at a time, I'd still be doing 8 sections in under a second.

[ximion]

Why it happens was completely obvious to me ^^
We might soon have a solution to not reprocess a suite if it hasn't changed, which will make this easier.
As for the patch, it looks like my mind had switched off there already. I wrote a bigger patch and last time reverted it to the smaller one because I though that was easier...
Anyway, the bigger patch is now in Git and wants to be tested ^^

[neilmayhew]

I'm happy to test, but I'm not seeing any new commits yet. Did you forget to push?

[ximion]

I just pushed something else, so the answer is: Yes, maybe ^^

[neilmayhew]

It looks good so far, but I need to look at the results a bit more and I've run out of time for today. I'll carry on next week.

[neilmayhew]

I tested bffb451 and it all seems good. Thanks for fixing this.

[neilmayhew]

BTW, I think you could delete the wording about being too fast now:

// we were too fast! - add the new data to this point in time
logDebug ("Attempted to add statistics at the exact same time when we have already added some. We are suspiciously fast...");

If the data is already cached, we should be fast.