how to run sequence_classifier method with pre-trained logits

[LiweiPeng]

Hi,

During my PET training, I found it is helpful to tune the last sequence classifier stage individually. What I plan to do is: 1) run PET method as normal; 2) In separate programs, load the merged unlabeled_logits.txt and tune sequence classifier parameters. I'd like to do step2 individually without reruning full PET.

I hit an issue when I tried to do the above step 2. While expected accuracy for my dataset is ~50%, I only got 1.5%. I used the same parameters for sequence classifier as I did for PET.

The following is what I did:

1. change this cli.py line to use_logits=True;
2. add an option in modeling.py train_classifier() so that it will load given logits file (unlabeled_logits.txt) like this line, then assign the logits to unlabeled data.

My questions are: what am I missing? What's the proper way to run sequence classifier with pre-trained/merged logits file?

Thanks.

[timoschick]

Hi @LiweiPeng, do things work as expected if you use the regular PET pipeline or do you also get an accuracy of 1.5% in this case? (If it's the latter, what accuracies do your individual pattern/verbalizer pairs achieve?)

The changes you've performed seem reasonable to me, I can't directly see anything that would be missing. Can you provide me with the exact changes that you've made?

There should also be an easier way to achieve what you want: You can simply rerun full PET because it will check whether the individual models already exist and if so, it will not retrain them (see this line), so the only thing that is recomputed is the merged logits file, but that should be very fast. You would also have to either delete or rename the directory containing the final model after each run as it will otherwise also not be retrained because of the very same line.

[LiweiPeng]

Hi @timoschick ,

Thanks for the quick response and suggested better method to run sequence classifier stage only.

The following are more details about my tests:

• labels: total 65 labels; each label has 2 tokens. verbalize() just return the label.
• dataset size: train=536, dev=536, unlabeled=536. these are all different data.
• MultiMaskTaskHelper is used
• 1 pattern is defined, pet_repetitions=1.

To answer your question, using the full PET method from scratch, final accuracy is 50.1%. Because only 1 PVP pattern is used and I didn't tune the pattern yet, any score around 50% is good enough to me.

Using the method you suggested where PET will skip pretrained patterns, the final accuracy is also very bad, around 1.3%.

Another test I did: run the sequence_classifier method using the PET code in the github (not my hack in my original post), which will use train dataset instead of unlabeled dataset, the accuracy is ~52%.

The p0-i0/results.json:
{"train_set_before_training": 0.014925373134328358, "global_step": 671, "average_loss": 2.196320776212766, "train_set_after_training": 0.9253731343283582, "test_set_after_training": {"acc": 0.48507462686567165}}

The following are logs from full PET method from scratch:

...<PET train and eval steps>...
modeling - --- RESULT (pattern_id=0, iteration=0) ---
modeling - {'acc': 0.48507462686567165}
modeling - === OVERALL RESULTS ===
modeling - acc-p0: 0.48507462686567165 +- 0
modeling - acc-all-p: 0.48507462686567165 +- 0
merge_logits logits_dir=checkpoint
modeling - Found the following 1 subdirectories: ['p0-i0']
modeling - File checkpoint/p0-i0/results.txt: Score = 0.014925373134328358, #Logits = 536, #Labels = 65
modeling - Got 536 logits from file checkpoint/unlabeled_logits.txt

...<SC train and eval steps>

modeling - --- RESULT (pattern_id=0, iteration=0) ---
modeling - {'acc': 0.5055970149253731}
modeling - === OVERALL RESULTS ===
modeling - acc-p0: 0.5055970149253731 +- 0
modeling - acc-all-p: 0.5055970149253731 +- 0

The following are logs following your suggested method, let PET to skip pretrained patterns:

...
modeling - Path checkpoint/p0-i0 already exists, skipping it...
modeling - === OVERALL RESULTS ===
merge_logits logits_dir=checkpoint
modeling - Found the following 1 subdirectories: ['p0-i0']
modeling - File checkpoint/p0-i0/results.txt: Score = 0.014925373134328358, #Logits = 536, #Labels = 65
modeling - Got 536 logits from file checkpoint/unlabeled_logits.txt

...<SC train and eval steps>

modeling - --- RESULT (pattern_id=0, iteration=0) ---
modeling - {'acc': 0.013059701492537313}
modeling - === OVERALL RESULTS ===
modeling - acc-p0: 0.013059701492537313 +- 0
modeling - acc-all-p: 0.013059701492537313 +- 0

[timoschick]

That's really odd -- I don't really have any good explanation for why this might happen at the moment. Could you share your source code (your PVP, Task and other modifications you've made) with me so that I can take a closer look? (However, I'll be on vacation starting tomorrow until October 4th so I might take some time to reply).

[LiweiPeng]

@timoschick Thanks for the help. I sent an email to you including all my code, dataset and full logs.

[LiweiPeng]

@timoschick Can you provide any updates on this issue? If the behavior I mentioned in my original message expected? Thanks.

[LiweiPeng]

I reran same tests with the 'yahoo' dataset used in original paper. The issue was not reproduced. So this error is from my dataset, not from PET. So I'll close this issue.