Asking about YelpChi dataset

[minhduc0711]

Hi, I'm currently reading the GraphConsis paper and I saw table 2:

I did notice that the features used in the paper are extracted using word2vec instead of BoW, but should these results stay roughly the same for both feature types? At the moment I couldn't reproduce these results even with the simplest Logistic Regression.

I got AUC of only 0.5 with the default sklearn LogisticRegression, using 80% training data.

[YingtongDou]

Hi,

Thanks for your question. The node feature used in the paper is actually the BoW feature. The word2vec has a slightly worse performance than the BoW during the experiments.

I just reproduced the same results of LR in Table 2 using the scikit-learn package on my computer. My scikit-learn LogisticRegression parameters are: solver='lbfgs', multi_class='multinomial', max_iter=1000.

The AUC = 0.5 indicates your classifier is not optimized or overfitting the data. To avoid overfitting, you can undersample the negative instances to balance the class distribution in the training data. You may use this approach to train the GNN since it is prone to overfit the training set when its class distribution is extremely imbalanced.

BTW, the text embedding is not informative for spam detection task on the Yelp dataset. We have another YelpChi graph with handcrafted features that have better performance. You can find this new dataset from this repo. We will update this new dataset to the DGFraud repo in the future.

Let me know if you have further questions.

[minhduc0711]

thanks @YingtongDou for the explanation, I set class_weights='balanced' in LogisticRegression and got AUC of 0.62 now.