test_run_glue_with_pabee failing

[sshleifer]

examples/bert-loses-patience/test_run_glue_with_pabee.py::PabeeTests::test_run_glue

https://app.circleci.com/pipelines/github/huggingface/transformers/10373/workflows/0c9f2e61-2732-4857-84f0-71b59ddf10a9/jobs/71646

@JetRunner

[JetRunner]

@sshleifer Thanks for the issue. I noticed that this test fails sometimes. Do you have any idea how long has this problem been? Is it broken at the beginning or recently?

[sshleifer]

Started breaking within the last few days. It is breaking fairly consistently at this point.

[sshleifer]

It's I guess just flaky, for example it's not broken on master right now.

[sshleifer]

But it's not really a flaky "type of error". It's hitting IndexError on an embedding lookup.

IndexError: index out of range in self

[sgugger]

It also fails with the assert: self.assertGreaterEqual(value, 0.75) in my case (three times out of four right now).