What do you mean by " We apply DIRECTPROBE on the training and test set separately"?

[eunjiinkim]

Hi,

in your paper, A Closer Look at How FIne-tuning Changes BERT, it is written that "We apply DIRECTPROBE on the
training and test set separately" in section 4.1.
For DirectProbing, we need train and test set. Then, does it mean that you split training set into train/test and test set into train/test set too?
Or just to use training as a test set too?

Thanks. :D

[flyaway1217]

For DirectProbe, we do not need to differentiate the training or test set. What DirectProbe do is it takes into a labeled dataset and produces a set of clusters. It does not know about training or test set.
In Section 4.1, we apply DirectProbe to the training and test sets to show how fine-tuning changes the geometry of the embeddings, i.e., fine-tuning diverges the training and test set.
I hope that clarifies your questions.
Thanks!

[eunjiinkim]

@flyaway1217

Thanks! Now I understand that line.
Then train.txt and test.txt in config file are probed separately?

entities_path = ${common}/entities/train.txt
test_entities_path = ${common}/entities/test.txt

But when I runned the code with train.txt and test.txt only one per result text files are saved like below.
Also I found that I should set both train and test files.
How can I interpret the results? Or should I set both train and test files as the same one?

[eunjiinkim]

Oh, I found that the current codes actually probe only entities_path and embeddings_path and do not probe test files, right?

[flyaway1217]

Yes. You are correct. Every time, DirectProbe only clusters for one dataset. That test_entities_path is something from the previous version. We do not use it in the paper "A Closer Look at How FIne-tuning Changes BERT."

[flyaway1217]

By the way, You may want to pull the latest version. We recently fixed a minor bug in the code.

[flyaway1217]

@flyaway1217

Thanks! Now I understand that line. Then train.txt and test.txt in config file are probed separately?

entities_path = ${common}/entities/train.txt
test_entities_path = ${common}/entities/test.txt

But when I runned the code with train.txt and test.txt only one per result text files are saved like below. Also I found that I should set both train and test files. How can I interpret the results? Or should I set both train and test files as the same one?

You need to provide something for the test_entities_path, but it will not be used. The output is the results of entities_path.

[eunjiinkim]

Thanks ! I will update the codes to the latest version.
Thank you for your clear comments. :)

[eunjiinkim]

@flyaway1217
Hi, I have one more question. When I run the codes with my own data which has 4 labels and about 7600 entities, it takes too much time (up to 4-5 hours) with the error below.
UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak. "timeout or by a memory leak.", UserWarning

Do you have any solution to this problem? Or should I just wait till the end of the clustering?

[flyaway1217]

Sometimes I had the same warning. Usually, I just wait until the end.
If Directprobe takes a long time to finish, your representation is non-linear for the given task.
One thing you could try is to change rate in the config.ini file. It controls the size of the step during the clustering process. I suggest that you can try between [0.05, 0.2].
In my own experiments, we don't have many non-linear cases. But I can say that 4-5 hours is within the normal range.

[flyaway1217]

Also, the time depends on how many CPU you have because Directprobe uses multiple processes to do the linearity check. More CPU usually means faster clustering.