New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ctc_beam_search_decoder()'s log_probabilities holds invalid values #6034

Closed
kdavis-mozilla opened this Issue Dec 2, 2016 · 28 comments

Comments

Projects
None yet
@kdavis-mozilla
Contributor

kdavis-mozilla commented Dec 2, 2016

Environment info

Operating System: OS X 10.11.6
TF Version: 0.10.0rc0 (No GPU)

Example

Ran log_probabilities op created from

decoded, log_probabilities = ctc_ops.ctc_beam_search_decoder(logits, \
                                                             seq_length, \
                                                             beam_width=beam_width, \
                                                             top_paths=top_paths, \
                                                             merge_repeated=False)

The decoded result is as expected. However, the log_probabilities contains positive values which can not be log probabilities. For example, with batch size 4 and top_paths=10 the log_probabilities printout is as follows

[[ 3.85424066 -1.97321272 -1.99056399 -2.18253303 -2.18592954 -2.40727925
  -2.87798476 -2.88267159 -2.94563317 -2.94854331]
 [ 3.85424066 -1.97321272 -1.99056399 -2.18253303 -2.18592954 -2.40727925
  -2.87798476 -2.88267159 -2.94563317 -2.94854331]
 [ 3.85424066 -1.97321272 -1.99056399 -2.18253303 -2.18592954 -2.40727925
  -2.87798476 -2.88267159 -2.94563317 -2.94854331]
 [ 3.85424066 -1.97321272 -1.99056399 -2.18253303 -2.18592954 -2.40727925
  -2.87798476 -2.88267159 -2.94563317 -2.94854331]]

Other attempted solutions

None

Logs or other output that would be helpful

Link to the entire code in context[1]

@ebrevdo

This comment has been minimized.

Contributor

ebrevdo commented Dec 3, 2016

I may not get to this before January.

@kdavis-mozilla

This comment has been minimized.

Contributor

kdavis-mozilla commented Dec 11, 2016

Is the returned log_probabilities supposed to represented the logs of a probabilities, as documented[1], or does it represent un-normalized logs of a probabilities?

@ebrevdo

This comment has been minimized.

Contributor

ebrevdo commented Dec 11, 2016

Should be normalized, I'll have to trace this exact example to see what's going on

@vrenkens

This comment has been minimized.

vrenkens commented Jan 12, 2017

I have the same problem, with beam search I get positive scores but with greedy I get negative ones

@kdavis-mozilla

This comment has been minimized.

Contributor

kdavis-mozilla commented Jan 26, 2017

@ebrevdo Has there been any progress on this?

@ebrevdo

This comment has been minimized.

Contributor

ebrevdo commented Jan 26, 2017

Please provide a small example input that causes the issue so I can debug

@vrenkens

This comment has been minimized.

vrenkens commented Jan 26, 2017

I am not really familiar with the code, but is it possible that it is caused by line 183 in tensorflow/tensorflow/core/util/ctc/ctc_beam_search.h: (*scores)(b, i) = -beam_log_probabilities[i]; ?

@kdavis-mozilla

This comment has been minimized.

Contributor

kdavis-mozilla commented Jan 30, 2017

@ebrevdo Until I get time to write a smaller, example one can look at ctc_decoder_ops_test.py

Documentation of ctc_beam_search_decoder states

...
Returns:

A tuple (decoded, log_probabilities) where

decoded: A list of length top_paths, where decoded[j] is a SparseTensor ...
log_probability: A float matrix (batch_size x top_paths) containing sequence log-probabilities.

However, for ctc_decoder_ops_test.py the expected results, the log_prob_truth values, are

    log_prob_truth = np.array(
        [
            0.584855,  # output beam 0
            0.389139  # output beam 1
        ],
        np.float32)[np.newaxis, :]

These are not log probabilities as they are positive.

@Garvys

This comment has been minimized.

Contributor

Garvys commented Mar 10, 2017

@ebrevdo Is there any progress on this issue ? I have this problem with TF 1.0

@tychotax

This comment has been minimized.

tychotax commented Apr 18, 2017

@ebrevdo same issue here!

@qiqiguaitm

This comment has been minimized.

qiqiguaitm commented Jun 7, 2017

same issue

@BradNeuberg

This comment has been minimized.

BradNeuberg commented Jan 4, 2018

This issue actually prevented a project here at Dropbox from migrating our training pipeline from Torch to TensorFlow, unfortunately causing us to stay with Torch.

@ebrevdo

This comment has been minimized.

Contributor

ebrevdo commented Jan 4, 2018

@kdavis-mozilla in your initial report, what were the seq_len values passed in?

@kdavis-mozilla

This comment has been minimized.

Contributor

kdavis-mozilla commented Jan 5, 2018

@ebrevdo Sorry but I don't even know anymore; it was over a year ago.

We've since created our own custom ctc_beam_search_decoder operator extending the TF operator with KenLM. We use that now.

As far a I remember, the problem is still there. The relative ordering of results via their log_probabilities is fine, but the absolute log_probability values are not.

@ryanleary

This comment has been minimized.

ryanleary commented Jan 5, 2018

The problem is still there. It's related to the subtraction of the maxCoeff from the input and then the subsequent negation of the log probs. I can submit a PR with the fix, but don't have time to workthrough the broken test cases.

@ebrevdo

This comment has been minimized.

Contributor

ebrevdo commented Jan 5, 2018

@chenjiasheng

This comment has been minimized.

chenjiasheng commented Jan 24, 2018

@ryanleary
Can we fix this issue in Python side with touching C++?

top_paths=3, beam_width=400, I get these log-probs:

array([[ 0.09569344, -2.35573864, -3.55027127],
       [ 0.31278262, -0.11559263, -0.22166124],
       [ 0.01575552, -0.69348544, -0.84170169],
       [ 0.08385472, -0.10281177, -1.23370361],
       [ 1.34617054,  1.31623089, -0.14258756],
       [ 0.1471574 , -0.13724545, -0.34559944],
       [ 0.3714101 ,  0.02229187, -0.05457714],
       [ 0.72338897, -0.42095903, -0.4398351 ],
       [ 0.66841668, -0.19791916, -0.73667264],
       [ 0.11281018, -0.14468569, -2.61785507],
       [ 0.02421612, -0.29929882, -0.64302784],
       [ 0.01053159, -0.2815102 , -1.38790202],
       [ 0.03269538, -0.30419388, -0.4501299 ],
       [ 0.00933365, -0.72536844, -1.57513046],
       [ 0.40954223,  0.0353958 , -1.72318888],
       ............
      ], dtype=float32)
@ryanleary

This comment has been minimized.

ryanleary commented Feb 22, 2018

@ebrevdo yes, I think something like that needs to occur (to the extent that it can be done). What underflow/overflow issues have you experienced? Is that still really an issue even though we're in the log prob space?

@shokou-chin

This comment has been minimized.

shokou-chin commented Jun 24, 2018

Is there any progress on this issue?

@igormq

This comment has been minimized.

Contributor

igormq commented Jul 10, 2018

Up.

@ebrevdo

This comment has been minimized.

Contributor

ebrevdo commented Jul 10, 2018

@igormq

This comment has been minimized.

Contributor

igormq commented Jul 11, 2018

What do you mean?

This issue has more than one year now, and as you said it is an important issue. Isn't there anyone here that can have a look into this?

@ebrevdo

This comment has been minimized.

Contributor

ebrevdo commented Jul 11, 2018

@tensorflowbutler

This comment has been minimized.

Member

tensorflowbutler commented Aug 31, 2018

Nagging Assignee @ebrevdo: It has been 34 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@ebrevdo

This comment has been minimized.

Contributor

ebrevdo commented Aug 31, 2018

I think this is fixed now?

@ebrevdo ebrevdo closed this Aug 31, 2018

@kdavis-mozilla

This comment has been minimized.

Contributor

kdavis-mozilla commented Aug 31, 2018

@ebrevdo What commit fixed this problem?

@ebrevdo

This comment has been minimized.

Contributor

ebrevdo commented Aug 31, 2018

@kdavis-mozilla

This comment has been minimized.

Contributor

kdavis-mozilla commented Aug 31, 2018

@ebrevdo Thanks for the info

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment