Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Full benchmark Updates #438

Merged
merged 3 commits into from
Mar 8, 2017
Merged

Conversation

lilleswing
Copy link
Member

qm7b, toxcast, and kaggle are currently broken and I cannot run them through end to end. I will file issues for each one with the stack trace.

We also have multiple results not hitting desired expectations. Below is the full list.

Dataset Model Split Metric Retrieved Value Desired Value
muv Multitask network Index splitting test_score 0.749443916693 0.764
muv logistic regression Index splitting test_score 0.756238252709 0.766
muv graph convolution Index splitting test_score 0.791876998348 0.823
pcba Multitask network Index splitting train_score 0.814176263345 0.826
pcba Multitask network Index splitting test_score 0.788107828923 0.802
clintox Multitask network Index splitting test_score 0.819248826291 0.83
clintox robust MT-NN Index splitting test_score 0.81514084507 0.827
nci graphconv regression Index splitting train_score 0.112239244388 0.123
nci graphconv regression Index splitting test_score 0.0464834859959 0.048
qm7b NN regression Index splitting train_score 0.876963119457 0.931
qm7b NN regression Index splitting test_score 0.790498748877 0.803
qm7b NN regression Index splitting train_score 0.878199577192 0.931
qm7b NN regression Index splitting test_score 0.791941826351 0.803
qm7b NN regression Index splitting train_score 0.879837090375 0.931
qm7b NN regression Index splitting test_score 0.792008993439 0.803
qm7 NN regression Index splitting test_score 0.97433488412 0.986
sampl NN regression Index splitting test_score 0.755635996 0.764
sampl NN regression Random splitting test_score 0.761702572858 0.83
sampl NN regression Scaffold splitting train_score 0.880153572737 0.891
nci graphconv regression Index splitting train_score 0.117258262877 0.123
nci graphconv regression Index splitting train_score 0.115851838028 0.123
nci graphconv regression Index splitting test_score 0.0463009131706 0.048
qm7 NN regression Index splitting test_score 0.958073353377 0.986
nci graphconv regression Random splitting train_score 0.106167884328 0.117
nci graphconv regression Random splitting test_score 0.0706557598031 0.076

@coveralls
Copy link

coveralls commented Mar 6, 2017

Coverage Status

Coverage remained the same at 85.447% when pulling ce95dbc on lilleswing:full-benchmark-cr into 10b6cd8 on deepchem:master.

@coveralls
Copy link

coveralls commented Mar 7, 2017

Coverage Status

Coverage decreased (-0.1%) to 85.326% when pulling 255d43a on lilleswing:full-benchmark-cr into 10b6cd8 on deepchem:master.

@rbharath
Copy link
Member

rbharath commented Mar 8, 2017

@lilleswing Thanks for the PR! We will need to find a systematic way of ensuring that the benchmarks here and MoleculeNet numbers match to within 1%. Could you open an issue with the table of outstanding failures? I'm going to go ahead and merge this PR in.

@rbharath rbharath merged commit 879ddaa into deepchem:master Mar 8, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants