How to use CTC loss function?

[F0REacH]

How to use CTC loss function?
I'm planning to extend my dataset with unsegmented data, but cannot figure out how to use ctc_error_evaluator and ctc_layer
model:

def stacked_gru_net_with_ctc(input_dim=24,
                     class_dim=133,
                     hid_dim=129,
                     stacked_num=3,
                     is_predict=False):
    """
    """
    assert stacked_num % 2 == 1

    linear = LinearActivation()

    data = data_layer("word", input_dim)
    fc1 = fc_layer(input=data, size=hid_dim, act=linear)
    gru1 = grumemory(input=fc1)

    inputs = [fc1, gru1]
    for i in range(2, stacked_num + 1):
        fc = fc_layer(input=inputs, size=hid_dim, act=linear)
        gru = grumemory(input=fc, reverse=(i % 2) == 0)
        inputs = [fc, gru]


    output = fc_layer(input=inputs, size=class_dim, act=SoftmaxActivation())

    if is_predict:
        outputs(output)
    else:
        # FIXME? #outputs(classification_cost(input=output, label=data_layer('label', class_dim), evaluator=ctc_error_evaluator))
        outputs(classification_cost(input=output, label=data_layer('label', class_dim), evaluator=precision_recall_evaluator))

dataprovider:

    settings.input_types = [
        dense_vector_sequence(24),
        integer_value_sequence(133)]

Also I have no idea how to feed labels to model if sequences are not segmented. With simple classification_cost each timestep has label, but how to use dataprovider with unsegmented sequences and CTC?
My current dataset is 180 examples, each is roughly 5000 timesteps (variable length). Each timestep is len=24 float vector labeled with one int label in range 0..132 (133 labels total).
Any example/suggestion would be highly appreciated. Thanks

[qingqing01]

You can use ctc_layer and ctc_error_evaluator as follows. Note, in consideration of the ‘blank’ label is needed by CTC, the size of ctc_layer and its input layer should be 134 (133 + 1). The 'blank' is the last category index by default, namely 133. We also integrate warp-CTC before, which may be merged later.

   ... # omit the previous content

   output = fc_layer(input=inputs, size=134, act=SoftmaxActivation())

   label = data_layer('label', class_dim)
   ctc = ctc_layer(input=output, label=label, 134)
   eval = ctc_error_evaluator(input=output, label=label)
   outputs(ctc)

[F0REacH]

@qingqing01 Many thanks.

[qingqing01]

@F0REacH I forget to remind you to update the code.

[F0REacH]

@qingqing01 I saw commit 9f3cbed fixed ctc issues. Nice work

[F0REacH]

Hi, @qingqing01
I've pulled the latest code and changed the model as you suggested:

def stacked_gru_net_with_ctc(input_dim=24,
                     class_dim=133,
                     hid_dim=129,
                     stacked_num=3,
                     is_predict=False):
    """
    """
    assert stacked_num % 2 == 1

    linear = LinearActivation()

    data = data_layer("word", input_dim)
    fc1 = fc_layer(input=data, size=hid_dim, act=linear)
    gru1 = grumemory(input=fc1)

    inputs = [fc1, gru1]
    for i in range(2, stacked_num + 1):
        fc = fc_layer(input=inputs, size=hid_dim, act=linear)
        gru = grumemory(input=fc, reverse=(i % 2) == 0)
        inputs = [fc, gru]

    output = fc_layer(input=inputs, size=class_dim+1, act=SoftmaxActivation())
    label = data_layer('label', class_dim)
    ctc = ctc_layer(input=output, label=label, size=class_dim+1)

    if is_predict:
        outputs(output)
    else:
        eval = ctc_error_evaluator(input=output, label=label)
        outputs(ctc)

But somehow started getting segmentation fault error:

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x0000000000b1ed66 in __gnu_cxx::__ops::_Iter_less_iter::operator()<float*, float*> (this=0x7ffcb7b25980, __it1=0x21cc21000, __it2=0x21cc21004) at /opt/gcc/include/c++/4.9.4/bits/predefined_ops.h:42
42            { return *__it1 < *__it2; }
[Current thread is 1 (Thread 0x7f5fe144f800 (LWP 8836))]
(gdb) bt
#0  0x0000000000b1ed66 in __gnu_cxx::__ops::_Iter_less_iter::operator()<float*, float*> (this=0x7ffcb7b25980, __it1=0x21cc21000, __it2=0x21cc21004) at /opt/gcc/include/c++/4.9.4/bits/predefined_ops.h:42
#1  0x0000000000b1e9ed in std::__max_element<float*, __gnu_cxx::__ops::_Iter_less_iter> (__first=0x21cc21004, __last=0x21cc21218, __comp=...) at /opt/gcc/include/c++/4.9.4/bits/stl_algo.h:5474
#2  0x0000000000b1e84c in std::max_element<float*> (__first=0x21cc21000, __last=0x21cc21218) at /opt/gcc/include/c++/4.9.4/bits/stl_algo.h:5497
#3  0x0000000000b1d1dc in paddle::CTCErrorEvaluator::bestLabelSeq (this=0x13a573f0) at /home/foreach/SOFT/BAIDU/PADDLE/Paddle/paddle/gserver/evaluators/CTCErrorEvaluator.cpp:52
#4  0x0000000000b1dc29 in paddle::CTCErrorEvaluator::editDistance (this=0x13a573f0, output=0x219046ce8, numTimes=120314, numClasses=134, labels=0x2049e5928, labelsLen=61) at /home/foreach/SOFT/BAIDU/PADDLE/Paddle/paddle/gserver/evaluators/CTCErrorEvaluator.cpp:175
#5  0x0000000000b1e152 in paddle::CTCErrorEvaluator::evalImp (this=0x13a573f0, arguments=std::vector of length 2, capacity 2 = {...}) at /home/foreach/SOFT/BAIDU/PADDLE/Paddle/paddle/gserver/evaluators/CTCErrorEvaluator.cpp:212
#6  0x0000000000affcae in paddle::Evaluator::eval (this=0x13a573f0, nn=...) at /home/foreach/SOFT/BAIDU/PADDLE/Paddle/paddle/gserver/evaluators/Evaluator.cpp:32
#7  0x0000000000b1e23e in paddle::CTCErrorEvaluator::eval (this=0x13a573f0, nn=...) at /home/foreach/SOFT/BAIDU/PADDLE/Paddle/paddle/gserver/evaluators/CTCErrorEvaluator.cpp:221
#8  0x0000000000ad6e4e in paddle::CombinedEvaluator::eval (this=0x13a57340, nn=...) at /home/foreach/SOFT/BAIDU/PADDLE/Paddle/paddle/gserver/gradientmachines/NeuralNetwork.cpp:321
#9  0x0000000000ad4f44 in paddle::NeuralNetwork::eval (this=0x136f0e70, evaluator=0x13a57340) at /home/foreach/SOFT/BAIDU/PADDLE/Paddle/paddle/gserver/gradientmachines/NeuralNetwork.cpp:379
#10 0x0000000000bc461e in paddle::TrainerInternal::trainOneBatch (this=0x7ffcb7b26a30, batchId=0, dataBatch=...) at /home/foreach/SOFT/BAIDU/PADDLE/Paddle/paddle/trainer/TrainerInternal.cpp:143
#11 0x0000000000bbe532 in paddle::Trainer::trainOnePass (this=0x7ffcb7b26980, passId=0) at /home/foreach/SOFT/BAIDU/PADDLE/Paddle/paddle/trainer/Trainer.cpp:434
#12 0x0000000000bbd1e0 in paddle::Trainer::train (this=0x7ffcb7b26980, numPasses=100000) at /home/foreach/SOFT/BAIDU/PADDLE/Paddle/paddle/trainer/Trainer.cpp:280
#13 0x00000000009f2761 in main (argc=12, argv=0x7ffcb7b27518) at /home/foreach/SOFT/BAIDU/PADDLE/Paddle/paddle/trainer/TrainerMain.cpp:100
(gdb) frame 3
#3  0x0000000000b1d1dc in paddle::CTCErrorEvaluator::bestLabelSeq (this=0x13a573f0) at /home/foreach/SOFT/BAIDU/PADDLE/Paddle/paddle/gserver/evaluators/CTCErrorEvaluator.cpp:52
52                                            acts + (i + 1) * numClasses_) -
(gdb) info locals
i = 117089
path = std::vector of length 117089, capacity 131072 = {118, 60, 7, 7, 97, 40, 53, 40, 53, 53, 53, 7, 7, 7, 7, 7, 44, 44, 132, 7, 7, 53, 53, 53, 53, 53, 53, 53, 53, 97, 97, 7, 53, 53, 40, 53, 53, 17, 7, 7, 44, 44, 53, 53, 53, 22, 7, 44, 44, 17, 7, 87, 87, 53, 53, 53, 53, 53, 53, 53, 
  53, 53, 53, 53, 53, 53, 7, 51, 53, 53, 53, 129, 129, 48, 48, 48, 48, 48, 112, 112, 2, 47, 31, 60, 48, 48, 48, 20, 60, 60, 60, 60, 60, 6, 48, 48, 48, 48, 39, 39, 60, 127, 48, 127, 127, 127, 127, 127, 127, 127, 127, 127, 127, 127, 127, 127, 127, 6, 127, 127, 127, 127, 127, 127, 60, 
  127, 127, 60, 6, 38, 118, 38, 84, 84, 84, 84, 39, 39, 127, 127, 127, 127, 127, 2, 54, 54, 47, 47, 47, 48, 48, 48, 48, 48, 15, 48, 48, 48, 47, 47, 47, 48, 47, 48, 48, 48, 48, 48, 47, 48, 48, 48, 112, 97, 97, 6, 39, 6, 21, 6, 21, 39, 127, 48, 48, 54, 54, 48, 48, 127, 127, 127, 127, 
  127, 127, 127, 127, 127, 127, 127...}
acts = 0x219046ce8
(gdb) p numClasses_
$1 = 134
(gdb) p numTimes_
$2 = 120314

[qingqing01]

Hi, please refer to the changes in paddle/gserver/evaluators/CTCErrorEvaluator.cpp at this pull https://github.com/baidu/Paddle/pull/82/files

[F0REacH]

Thanks, @qingqing01 with pull #82 all is OK.