Worse results with Banking Dataset

[rajat-tech-002]

@Dejiao2018
Hy, the idea in the paper is quite good.
But I am not able to get good results with a different dataset. For eg. Banking Dataset (77 clusters) (https://arxiv.org/pdf/2003.04807.pdf)

Convergence Criteria: Percentage Change in Prediction in Subsequent Iterations < 0.1 %
Using the same available code.

Initial Results Before Training: Kmeans : ACC: 42.4 NMI: 60.95 ARI: 27.49

After Training: Kmeans : ACC: 13.0 NMI: 30.32 ARI: 5.2
After Training: Cluster Model: ACC: 11.4 NMI: 26.12 ARI: 4.7

Over subsequent iterations, the results get worse.

[rajat-tech-002]

Hi @Dejiao2018 , Any update?
I can provide my training dataset file which contains orig_text and it's 2 augmentations generated using the contextual augmentor. (https://github.com/makcedward/nlpaug)
What could be the reason for these results?
Does it mean that SCCL approach may not work with 1 domain (eg Banking/Overlapping Classes) dataset?

[Dejiao2018]

Hi @rajat-tech-002 ,

Thank you for the question. A couple suggestions:

1. turn off the clustering head, optimize the contrastive loss only and track the clustering performance over the learning process (see every 50-100 iterations). If the clustering performance gets worse, then that basically indicates contrastive learning with your augmented data may not work in this setting. To further validate the effectiveness of your augmentation set, using dropout as augmentation (forwarding the same text twice, see the SimCSE paper), if either contrastive learning / SCCL works in this setting, then the problem probably is caused by the effectiveness of your augmentaton set.

2. if contrastive learning alone promotes better clustering performance, then tracking the statistics of the cluster head output, i.e., the entropy and conditional entropy of the softmax output. By doing so, you can debug whether the clustering head works in the expected way.

3. try a different backbone, e.g., the backbone pretrained using PairSupCon (https://arxiv.org/abs/2109.05424, will release our code soon), or the SimCSE checkpoints.