Use threading for PPMd8 decoders

[miurahr]

Partial fix for #28

Todo:

• support threading ppmd8 decode CPython on linux/mac and pthread
• support threading ppmd8 decode PyPy-CFFI on linux/mac and pthread
• support threading ppmd8 decode CPython on Windows

Issues:

• Abort on PyPy when some conditions

Signed-off-by: Hiroshi Miura miurahr@linux.com

[miurahr]

@cielavenir could you mind to review changes, test it, and help the progress?

[cielavenir]

@miurahr it is too large to review, but at least tests are working.

[miurahr]

The core functions to be reviewed are three in lib2/Ppmd8Tdecoder.c :

Byte TReader(const void *p)
is called from PPMd library under lib/ folder. It send signal empty when buffer exhausted, and wait signal notEmpty.

static void * Ppmd8T_decode_run(void *p)

is a function that run in the thread. the core logic comes from src/ext/_ppmdmodule.c L1440-L1470 and protect critical part with mutex.

int Ppmd8T_decode()
is a function that start and handle thread, and return when input buffer is empty.
It also handles signals and detect state of _run function.

[miurahr]

There are several space to be improved.

[miurahr]

Windows fatal exception: access violation errors are reproduced on Windows test, there may be a bug but it is too difficult to debug from log.

[cielavenir]

Actually I have an issue:

I found that PPMD_pthread_cond_wait1(&inEmpty, &mutex) could fail to receive inEmpty signal. I have not investigated deeply but it seems like that signal was sent during pthread_cond_timedwait's relocking (here relocking would be done because of pthread_cond_wait(&notEmpty, &mutex)).

This happens because pthread_cond_timedwait is not FIFO, so I wrote simple implementation in cielavenir@f224a04 (part of UseEndmarkProperly2).

[cielavenir]

Another question: did you test FFI version? The test does not run here.

Inserting cielavenir@43791a5 worked.

[miurahr]

I found that PPMD_pthread_cond_wait1(&inEmpty, &mutex) could fail to receive inEmpty signal. I have not investigated deeply but it seems like that signal was sent during pthread_cond_timedwait's relocking (here relocking would be done because of pthread_cond_wait(&notEmpty, &mutex)).

This happens because pthread_cond_timedwait is not FIFO, so I wrote simple implementation in cielavenir/pyppmd-py2@f224a04 (part of UseEndmarkProperly2).

Thank you for the suggestion!
I don't prefer global variable other than mutex lock one, so I'd like to rewrite based on your suggestion.

[cielavenir]

@miurahr see 7e707cc and 92d52df

squashing these two commits will become this (collapsed)

diff --git a/lib2/Ppmd8Tdecoder.c b/lib2/Ppmd8Tdecoder.c
index c3c257b..27717f3 100644
--- a/lib2/Ppmd8Tdecoder.c
+++ b/lib2/Ppmd8Tdecoder.c
@@ -5,12 +5,12 @@
 #include "Ppmd8Tdecoder.h"
 
 PPMD_pthread_mutex_t mutex;
-PPMD_pthread_cond_t finished, inEmpty, notEmpty;
+PPMD_pthread_cond_t addInput, notEmpty;
 
 Byte TReader(const void *p) {
     BufferReader *bufferReader = (BufferReader *)p;
     if (bufferReader->inBuffer->pos == bufferReader->inBuffer->size) {
-        PPMD_pthread_cond_signal(&inEmpty);
+        PPMD_pthread_cond_signal(&addInput);
         PPMD_pthread_cond_wait(&notEmpty, &mutex);
     }
     return *((const Byte *)bufferReader->inBuffer->src + bufferReader->inBuffer->pos++);
@@ -18,8 +18,7 @@ Byte TReader(const void *p) {
 
 Bool Ppmd8T_decode_init() {
     PPMD_pthread_mutex_init(&mutex, NULL);
-    PPMD_pthread_cond_init(&finished, NULL);
-    PPMD_pthread_cond_init(&inEmpty, NULL);
+    PPMD_pthread_cond_init(&addInput, NULL);
     PPMD_pthread_cond_init(&notEmpty, NULL);
     return True;
 }
@@ -69,6 +68,7 @@ Ppmd8T_decode_run(void *p) {
     PPMD_pthread_mutex_lock(&mutex);
     args->result = result;
     args->finished = True;
+    PPMD_pthread_cond_signal(&addInput);
     PPMD_pthread_mutex_unlock(&mutex);
     return NULL;
 }
@@ -101,6 +101,7 @@ int Ppmd8T_decode(CPpmd8 *cPpmd8, OutBuffer *out, int max_length, ppmd8_args *ar
             return -2;  // error
         }
     }
+#if 0
     while(True) {
         PPMD_pthread_mutex_lock(&mutex);
         if (PPMD_pthread_cond_wait1(&inEmpty, &mutex) == 0) {
@@ -114,6 +115,13 @@ int Ppmd8T_decode(CPpmd8 *cPpmd8, OutBuffer *out, int max_length, ppmd8_args *ar
         }
         PPMD_pthread_mutex_unlock(&mutex);
     }
-    PPMD_pthread_join(args->handle, NULL);
-    return args->result;
+#endif
+    PPMD_pthread_mutex_lock(&mutex);
+    PPMD_pthread_cond_wait(&addInput, &mutex);
+    PPMD_pthread_mutex_unlock(&mutex);
+    if (args->finished) {
+        PPMD_pthread_join(args->handle, NULL);
+        return args->result;
+    }
+    return 0;
 }

[miurahr]

I'd like to handle pypy3 case, infinite loop to wait thread finish, on another issue tracker.

[cielavenir]

maybe please merge this as it is and I'll revisit my UseEndmarkProperly branch later