Implementation of Weighted CRF Tagger (handling unbalanced datasets) #5676

eraldoluis · 2022-06-22T21:50:20Z

Closes #4619 .

Dependency of allennlp-models PR #341

Changes proposed in this pull request:

I implemented and experimentally compared three sample weighting strategies for CrfTagger.
The three strategies are implemented in the files allennlp/modules/conditional_random_field_**<strategy>**.py, where can be: wemission, wtrans, or lannoy.
I generalized the methods _input_likelihood(...) and _joint_likelihood(...) of the ConditionalRandomField class so that they now receive an argument with the transition weights. In that way, I could implement the two basic sample weighting strategies (wemission and wtrans) by just subclassing this class and weighting the corresponding weights (logits and transitions) in the forward(...) method before calling _input_likelihood(...) and _joint_likelihood(...). No modification was necessary to the basic algorithms in these two methods.
On the other hand, the strategy proposed by Lannoy et al. (suggestion by @dirkgr) needed a quite different implementation.

Before submitting

I've read and followed all steps in the Making a pull request
section of the CONTRIBUTING docs.
I've updated or added any relevant docstrings following the syntax described in the
Writing docstrings section of the CONTRIBUTING docs.
If this PR adds a new feature, I've added tests that sufficiently cover my new functionality.

After submitting

All GitHub Actions jobs for my pull request have passed.
codecov/patch reports high test coverage (at least 90%).
You can find this under the "Actions" tab of the pull request once the other checks have finished.

self.label_weights is now created as a parameter so that it will be moved to GPU whenvever the model moves.

epwalsh · 2022-06-30T16:13:31Z

Hi @eraldoluis, thanks for this! I may not have time for a thorough review this week but this will be a priority next week.

eraldoluis · 2022-07-01T06:55:12Z

Hi @eraldoluis, thanks for this! I may not have time for a thorough review this week but this will be a priority next week.

Thank you, @epwalsh !

epwalsh

Hey @eraldoluis, this looks really great! This is my first pass through (I still need to go through some of the code in more detail to get my head around it), but I just have some minor comments. In addition to the comments below, I will also say I think we should organize these modules into a common parent module. That is, create a new folder allennlp/modules/conditional_random_field/ and move the 3 implementations into there.

epwalsh · 2022-07-08T00:43:31Z

allennlp/modules/conditional_random_field_wemission.py

+        if label_weights is None:
+            raise ConfigurationError("label_weights must be given")
+
+        self.label_weights = torch.nn.Parameter(torch.Tensor(label_weights), requires_grad=False)


I think it might be better to use self.register_buffer() here instead of defining the weights as a parameter. That way we can be sure the label weights aren't passed to the optimizer.

https://discuss.pytorch.org/t/what-is-the-difference-between-register-buffer-and-register-parameter-of-nn-module/32723/11

epwalsh · 2022-07-08T00:44:44Z

allennlp/modules/conditional_random_field_wemission.py

+    This module uses the "forward-backward" algorithm to compute
+    the log-likelihood of its inputs assuming a conditional random field model.
+
+    See, e.g. http://www.cs.columbia.edu/~mcollins/fb.pdf


Could you add something about the weighting strategy here?

epwalsh · 2022-07-08T00:45:00Z

allennlp/modules/conditional_random_field_wtrans.py

+    This module uses the "forward-backward" algorithm to compute
+    the log-likelihood of its inputs assuming a conditional random field model.
+
+    See, e.g. http://www.cs.columbia.edu/~mcollins/fb.pdf


Should also have a note about the weighting strategy here.

epwalsh · 2022-07-08T00:46:08Z

allennlp/modules/conditional_random_field_lannoy.py

+    This module uses the "forward-backward" algorithm to compute
+    the log-likelihood of its inputs assuming a conditional random field model.
+
+    See, e.g. http://www.cs.columbia.edu/~mcollins/fb.pdf
+


Also would be good to have a note about the weight strategy here.

epwalsh · 2022-07-08T00:48:05Z

allennlp/modules/conditional_random_field_lannoy.py

+VITERBI_DECODING = Tuple[List[int], float]  # a list of tags, and a viterbi score
+
+
+def allowed_transitions(constraint_type: str, labels: Dict[int, str]) -> List[Tuple[int, int]]:


Is this any different from the same function in conditional_random_field.py?

I simply forgot to adapt this class. I think now it is much better.

epwalsh · 2022-07-08T00:48:19Z

allennlp/modules/conditional_random_field_lannoy.py

+    return allowed
+
+
+def is_transition_allowed(


Same question here: is any different from the original?

epwalsh · 2022-07-08T00:49:14Z

allennlp/modules/conditional_random_field_lannoy.py

+            for i, j in constraints:
+                constraint_mask[i, j] = 1.0
+
+        self._constraint_mask = torch.nn.Parameter(constraint_mask, requires_grad=False)


This should probably be a buffer as well, but I just realized this is how it's done in the original CRF module, so I guess I'm okay with this.

Ok. I didn't touch it. Although I agree with you that this should be a buffer as well.

Simplified Lannoy implementation.

eraldoluis · 2022-07-08T23:24:03Z

Thank you a lot @epwalsh for the effort you put on this.

I tried to address your first concerns. Let me know what you think about my changes.

I am looking forward to your feedback regarding the whole thing. Let me know if you have any questions. I will be happy to discuss this further if necessary.

epwalsh · 2022-07-12T22:20:02Z

Thanks for the quick responses/fixes! Changes look good. I should clarify what I meant by:

create a new folder allennlp/modules/conditional_random_field/ and move the 3 implementations into there.

Looks like you left allennlp/modules/conditional_random_field.py where it is, and then moved the weighted CRFs into allennlp/modules/conditional_random_field_weighted/. I'd rather have a single submodule (folder) called allennlp/modules/conditional_random_field/ with all of the CRFs (included the non-weighted base class).

epwalsh · 2022-07-13T00:14:26Z

I liked your blog post a lot by the way!

Renamed module allennlp.modules.conditional_random_field_weight to ...conditional_random_files

eraldoluis · 2022-07-13T21:50:54Z

Looks like you left allennlp/modules/conditional_random_field.py where it is, and then moved the weighted CRFs into allennlp/modules/conditional_random_field_weighted/. I'd rather have a single submodule (folder) called allennlp/modules/conditional_random_field/ with all of the CRFs (included the non-weighted base class).

Yes. I was unsure at first. But now I renamed the module to conditional_random_field and moved the original class to it. I also updated the changelog, which I had forgotten.

I also updated allennlp-models to reflect the new module organization. Unfortunately, I pushed first to the allennlp repository and the Model Tests failed (because allennlp-models was outdated). But these tests should pass now.

Let me know what do you think.

epwalsh

LGTM! I'll follow up with the PR in allennlp-models next

eraldoluis · 2022-07-14T21:00:24Z

Thank you very much, @epwalsh and @dirkgr ! This was my first contribution for an open source project and it was quite fun. I will definitely try it again soon. :)

eraldoluis added 6 commits June 22, 2022 22:33

Weighted CRF: scaled emission scores

6612457

Fixed bug in ConditionalRandomField

6ce3b15

self.label_weights is now created as a parameter so that it will be moved to GPU whenvever the model moves.

CRF weighting strategies

eed2eef

Weighted CRF: refactoring of three methods

595a488

Weighted CRF: refactoring of three methods

030da5c

Weighted CRF: black formatting

a7cf34c

eraldoluis mentioned this pull request Jun 22, 2022

Implementation of Weighted CRF Tagger (handling unbalanced datasets) allenai/allennlp-models#341

Merged

epwalsh self-assigned this Jun 30, 2022

Merge branch 'main' into weighted_crf

2a3e754

epwalsh reviewed Jul 8, 2022

View reviewed changes

eraldoluis added 2 commits July 9, 2022 00:38

Weighted CRF: moved classes to new module

4602ec0

Simplified Lannoy implementation.

formatting and type checking

a01d29f

eraldoluis added 3 commits July 13, 2022 23:09

Merge branch 'main' into weighted_crf

e3c5a18

Moved ConditionalRandomField to new module

236f654

Renamed module allennlp.modules.conditional_random_field_weight to ...conditional_random_files

Updated changelog

748bac6

epwalsh approved these changes Jul 14, 2022

View reviewed changes

epwalsh merged commit 5a3acba into allenai:main Jul 14, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implementation of Weighted CRF Tagger (handling unbalanced datasets) #5676

Implementation of Weighted CRF Tagger (handling unbalanced datasets) #5676

eraldoluis commented Jun 22, 2022 •

edited

epwalsh commented Jun 30, 2022

eraldoluis commented Jul 1, 2022

epwalsh left a comment

epwalsh Jul 8, 2022

eraldoluis Jul 8, 2022

epwalsh Jul 8, 2022

eraldoluis Jul 8, 2022

epwalsh Jul 8, 2022

eraldoluis Jul 8, 2022

epwalsh Jul 8, 2022

eraldoluis Jul 8, 2022

epwalsh Jul 8, 2022

eraldoluis Jul 8, 2022

epwalsh Jul 8, 2022

eraldoluis Jul 8, 2022

epwalsh Jul 8, 2022

eraldoluis Jul 8, 2022

eraldoluis commented Jul 8, 2022 •

edited

epwalsh commented Jul 12, 2022

epwalsh commented Jul 13, 2022

eraldoluis commented Jul 13, 2022

epwalsh left a comment

eraldoluis commented Jul 14, 2022

		VITERBI_DECODING = Tuple[List[int], float] # a list of tags, and a viterbi score


		def allowed_transitions(constraint_type: str, labels: Dict[int, str]) -> List[Tuple[int, int]]:

Implementation of Weighted CRF Tagger (handling unbalanced datasets) #5676

Implementation of Weighted CRF Tagger (handling unbalanced datasets) #5676

Conversation

eraldoluis commented Jun 22, 2022 • edited

Before submitting

After submitting

epwalsh commented Jun 30, 2022

eraldoluis commented Jul 1, 2022

epwalsh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eraldoluis commented Jul 8, 2022 • edited

epwalsh commented Jul 12, 2022

epwalsh commented Jul 13, 2022

eraldoluis commented Jul 13, 2022

epwalsh left a comment

Choose a reason for hiding this comment

eraldoluis commented Jul 14, 2022

eraldoluis commented Jun 22, 2022 •

edited

eraldoluis commented Jul 8, 2022 •

edited