CGNN results question

[sAviOr287]

Hi,

So I have tried to run the experiments again for the CGNN pairwise experiments.

And I can confirm to get the same results for the Multi, Gauss, Net, Tueb datasets in terms of AUPRC (using 12 different runs to ensemble)
AUPR: 0.95 MULTI
AUPR: 0.80 GAUSS
AUPR: 0.90 NET

However when I look at the acc ie. predicting the actual direction I get:
0.43, 0.46, 0.49 respectively.

I compute the acc by the score

for dataset_name in ['multi', 'gauss', 'net']:
	data, labels = load_dataset(dataset_name)
	res = genfromtxt('results/res2_{}.csv'.format(dataset_name), delimiter=',', skip_header=True)
	idx = 0
	acc = 0
	labels = labels.to_numpy()
	for data_ in res[:, 1]:
		if data_ < 0 and labels[idx] == -1:
			acc += 1
                        idx += 1 # EDIT
		elif data_ > 0 and labels[idx] == 1:
			acc += 1
                        idx += 1 # EDIT
		else:
			idx += 1

	acc /= (res.shape[0]-1)
	print(res.shape[0])
	print('{} ACC : {}'.format(acc, dataset_name))
	aupr, curve = precision_recall(labels[:res.shape[0]], res[:, 1])
	print('AUPR: {}'.format(aupr))

This method also gives me around 74% unweighted on Tueb dataset.

So my question is whether this is expected or whether i should be computing the acc differently or maybe even the ACC doesnt matter?

Thanks for the clarification in advance.

Best

[diviyank]

Hello, the results seem good, the accuracy should follow, could you join a sample of data_ ?
I think the predictions are not in the expected shape..

Best,

[sAviOr287]

res2_gauss.csv.zip
Hi

I have added the csv file that comes out after training the model.

Thanks for your help!

Best

[sAviOr287]

Here is also the way I loaded the data. I add this in cdt/data/loader.py

def load_ce_gauss(shuffle=False):
	dirname = os.path.dirname(os.path.realpath(__file__))

	data = read_causal_pairs('{}/resources/CE-Gauss_pairs.csv'.format(dirname), scale=False)
	labels = pd.read_csv('{}/resources/CE-Gauss_targets.csv'.format(dirname)).set_index('SampleID')

	if shuffle:
		for i in range(len(data)):
			if random.choice([True, False]):
				labels.iloc[i, 0] = -1
				buffer = data.iloc[i, 0]
				data.iloc[i, 0] = data.iloc[i, 1]
				data.iloc[i, 1] = buffer
	return data, labels


def load_ce_multi(shuffle=False):
	dirname = os.path.dirname(os.path.realpath(__file__))

	data = read_causal_pairs('{}/resources/CE-Multi_pairs.csv'.format(dirname), scale=False)
	labels = pd.read_csv('{}/resources/CE-Multi_targets.csv'.format(dirname)).set_index('SampleID')

	if shuffle:
		for i in range(len(data)):
			if random.choice([True, False]):
				labels.iloc[i, 0] = -1
				buffer = data.iloc[i, 0]
				data.iloc[i, 0] = data.iloc[i, 1]
				data.iloc[i, 1] = buffer
	return data, labels


def load_ce_net(shuffle=False):
	dirname = os.path.dirname(os.path.realpath(__file__))

	data = read_causal_pairs('{}/resources/CE-Net_pairs.csv'.format(dirname), scale=False)
	labels = pd.read_csv('{}/resources/CE-Net_targets.csv'.format(dirname)).set_index('SampleID')

	if shuffle:
		for i in range(len(data)):
			if random.choice([True, False]):
				labels.iloc[i, 0] = -1
				buffer = data.iloc[i, 0]
				data.iloc[i, 0] = data.iloc[i, 1]
				data.iloc[i, 1] = buffer
	return data, labels

[diviyank]

Hello,
Whoops I forgot to ask if you had the labels as well ?

[sAviOr287]

oh yeah I have

Archive.zip

which I downloaded from https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/3757KX

Thanks for the reply

Best

[diviyank]

Thanks for getting back to me quickly,

There seems to be an issue with your accuracy computation ; i got an accuracy of .72 on this dataset:

import pandas as pd
import numpy as np
from sklearn.metrics import average_precision_score, accuracy_score

preds = pd.read_csv('res2_gauss.csv')
labels = pd.read_csv('CE-Gauss_targets.csv')

print(labels.shape, preds.shape)
print(labels.columns, preds.columns)

# Returns :(300, 2) (300, 2)
# Returns : Index(['SampleID', 'Target'], dtype='object') Index(['SampleID', 'Predictions'], dtype='object')

average_precision_score(labels.Target, preds.Predictions) ## Equals to AUPR

# Returns :0.8027886920926466

preds.loc[preds.Predictions > 0, 'Predictions'] = 1
preds.loc[preds.Predictions < 0, 'Predictions'] = -1
accuracy_score(labels.Target,preds.Predictions)

# Returns :  0.7233333333333334

From my point of view, accuracy however might not be the best metric for evaluating causal algorithms: The confidence of an algorithm has to be taken into account, thus giving the possibility of not committing into a prediction if the prediction is not certain (Not answering is better that giving a wrong causal direction).

Best regards,
Diviyan

[sAviOr287]

Thanks a lot
Sorry, I was an idiot ...
I forgot to increment the idx variable

Thanks for your help

Sorry for the inconvenience

[diviyank]

No issues, glad I could help you!
I'll be closing this issue, have a good day !