-
Notifications
You must be signed in to change notification settings - Fork 26.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PhraseConstraints apearing only directly after input or at the end of the generated sentence #19070
Comments
cc @gante as well :) |
Hi @JoWohlen 👋 to confirm that I got the problem correctly -- the |
Yes that is correct |
Awesome, thank you for the clarification @JoWohlen 🙌 It helps to pinpoint the issue. I've added this issue to the list of |
You are welcome, and thanks for the great library! |
By accident I stumbled over what probably is the cause of all this. In #17814 a change was made to the constraint-beam-search. This change became active after v4.20.1 . Linked in the PR you can find another PR that adapts the tests to expect the faulty results (as in the issue description) |
Also @boy2000-007man, maybe you have a solution to this? |
@gante more generally should we maybe mark the disjunctive decoding as experimental and state that we don't actively maintain them? It's simply too time-consuming to look into this at the moment IMO |
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread. Please note that issues that do not follow the contributing guidelines are likely to be ignored. |
Would it be possible to keep this issue open? We are trying to improve output using constrained decoding and this issue prevents that. |
I am also interested to use this constrained text generation functionality which currently doesn't work anymore. |
Reopened (it's still on my generate task queue, which sadly is quite long) :) |
Looking forward to the solution. Btw, even using the version 4.20.1, which doesn’t have this issue, it also has the problem to use two or more words in force_word. |
@gante I would like to pick up this. Any pointers on where to start ? |
Hey @raghavanone 👋 Thank you for pitching in! I suggest opening two debugging sessions, one using v4.20 (where the output for this mode is correct) and the other using This is my go-to strategy for numerical problems in |
Hello everyone, Is there an update on this?
Weirdly, it works well when forcing chunks of two words, but fails when forcing chunks of > words. Here is an example to play around with:
when using
I think that if this could be fixed for 4.20.1, that would be an awesome next step. Additionally, it would be great to have the ability to define different constraints for each hypothesis in the input_ids. I suppose this line: https://github.com/huggingface/transformers/blob/v4.20.1/src/transformers/generation_beam_search.py#L477 should be changed so that the right constraint is returned according to the beam_idx |
Hi,I would love to work on this issue and fix the issue. |
@SHUBHAPRIYA95 feel free to open a PR and tag me :) |
I don't feel this is a bug. The 4.20.1 works because it inappropriately rewards constraints with |
System Info
transformers
version: 4.22.0Who can help?
@patrickvonplaten @Narsil @cwkeam
Information
Tasks
examples
folder (such as GLUE/SQuAD, ...)Reproduction
Overview
In the PR that introduced word constraints to the generation function we have an example script --> Example 2: A Mix of Strong Constraint and a Disjunctive Constraint.
Following up you see it slightly modified, but the modifications should not have an impact on the output
GPT2LMHeadModel
andGPT2Tokenizer
.to(torch_device)
for me to run the scriptself.....
ToDo
transformers==4.20.1
it works perfectly well4.20.1
it will not pass the assertionsExpected behavior
Problem
The constraining algorithm seems to be somewhat broken in versions above
4.20.1
For example on version
4.22
we the script generates the following the outputs:You can see that the constraints just get added to the end of the generated sentence. In fact, when trying around with constraints, I found out, that they are either placed right after the input:
--> example is made up to show what happens...
or at the end of the generated sentence:
4.20.1
they are just appended in a senseless manner?The text was updated successfully, but these errors were encountered: