Skip to content
Permalink
Browse files

wrapper for evaluation code

  • Loading branch information...
srijankr
srijankr committed Jul 15, 2019
1 parent fb38b6e commit 4c353504e580e26eda3ced6bd6d3e3b211f33ef7
@@ -0,0 +1,35 @@
#!/bin/bash

: '
This code evaluates the performance of the model for all epochs.
Then it runs the code that find the best validation epoch and uses it to calculate the performance of the model.
To run the code for interaction prediction on the reddit dataset:
$ ./evaluate_all_epochs.sh reddit interaction
To run the code for state change prediction on the reddit dataset:
$ ./evaluate_all_epochs.sh reddit state
'

network=$1
type=$2
interaction="interaction"

idx=0
while [ $idx -le 49 ]
do
echo $idx
if [ $type == "$interaction" ]; then
python2.7 evaluate_interaction_prediction.py --network $network --model jodie --epoch ${idx}
else
python2.7 evaluate_state_change_prediction.py --network $network --model jodie --epoch ${idx}
fi
(( idx+=1 ))
done


if [ $type == "$interaction" ]; then
python get_final_performance_numbers.py results/interaction_prediction_${network}.txt
else
python get_final_performance_numbers.py results/state_change_prediction_${network}.txt
fi
@@ -1,11 +1,12 @@
'''
This code evaluates the validation and test performance of the model trained in jodie.py.
This code evaluates the validation and test performance in an epoch of the model trained in jodie.py.
The task is: interaction prediction, i.e., predicting which item will a user interact with?
This has applications in recommender system and modeling network evolution.
How to run:
To calculate the performance for one epoch:
$ python evaluate_interaction_prediction.py --network reddit --model jodie --epoch 49
To calculate the performance for all epochs, use the bash file, evaluate_all_epochs.sh, which calls this file once for every epoch.
Paper: Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks. S. Kumar, X. Zhang, J. Leskovec. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2019.
'''

@@ -1,11 +1,12 @@
'''
This code evaluates the validation and test performance of the model trained in jodie.py.
This code evaluates the validation and test performance in an epoch of the model trained in jodie.py.
The task is: user state change prediction, i.e., when the state of a user changes from one to another, say normal to abnormal.
This has applications in detecting anomaly, fraud, churn, account compromise, and so on.
How to run:
To calculate the performance for one epoch:
$ python evaluate_state_change_prediction.py --network reddit --model jodie --epoch 49
To calculate the performance for all epochs, use the bash file, evaluate_all_epochs.sh, which calls this file once for every epoch.
Paper: Predicting Dynamic Embedding Trajectory in Temporal Interaction Networks. S. Kumar, X. Zhang, J. Leskovec. ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2019.
'''

@@ -219,6 +220,7 @@

print '\n\n*** Validation performance of epoch %d ***' % args.epoch
fw.write('\n\n*** Validation performance of epoch %d ***\n' % args.epoch)

for i in xrange(len(metrics)):
print(metrics[i] + ': ' + str(performance_dict['validation'][i]))
fw.write("Validation: " + metrics[i] + ': ' + str(performance_dict['validation'][i]) + "\n")
@@ -1,5 +1,5 @@
'''
This code is used to find the best validation epoch and use it to calculate the performance of the model.
This code is used to find the best validation epoch and to calculate the performance of the model.
How to run:
$ python get_final_performance_numbers.py results/interaction_prediction_reddit.txt
@@ -37,11 +37,19 @@

validation_performances = np.array(validation_performances)
test_performances = np.array(test_performances)
print "All validation performance numbers:", validation_performances
print "All test performance numbers:", test_performances

print "\n\n"
print "For file: %s" % fname
if "interaction" in fname:
metrics = ['Mean Reciprocal Rank', 'Recall@10']
else:
metrics = ['AUC']

print '\n\n*** For file: %s ***' % fname
best_val_idx = np.argmax(validation_performances[:,1])
print "Best validation performance:", validation_performances[best_val_idx]
print "Corresponding test performance:", test_performances[best_val_idx]
print "Best validation epoch: %d" % best_val_idx
print '\n\n*** Best validation performance (epoch %d) ***' % best_val_idx
for i in xrange(len(metrics)):
print(metrics[i] + ': ' + str(validation_performances[best_val_idx][i+1]))

print '\n\n*** Final model performance on the test set, i.e., in epoch %d ***' % best_val_idx
for i in xrange(len(metrics)):
print(metrics[i] + ': ' + str(test_performances[best_val_idx][i+1]))

0 comments on commit 4c35350

Please sign in to comment.
You can’t perform that action at this time.