Vocab is not accessible, it is in gs://t5-data

[cnut1648]

Hello, first of all, I want to say nice work!

When I want to reproduce your results on chemprot, I notice the following auth issue in the code

model.finetune(
    mixture_or_task_name="re_all",
    pretrained_model_dir=PRETRAINED_DIR,
    finetune_steps=FINETUNE_STEPS
)

2022-11-30 14:46:16.639835: W tensorflow/core/platform/cloud/google_auth_provider.cc:184] All attempts to get a Google authentication bearer token failed, returning an empty token. Retrieving token from files failed with "NOT_FOUND: Could not locate the credentials file.". Retrieving token from GCE failed with "FAILED_PRECONDITION: Error executing an HTTP request: libcurl code 6 meaning 'Couldn't resolve host name', error details: Could not resolve host: metadata".

Turns out that this is caused by not being able to find vocab which is in 'gs://t5-data/vocabs/cc_all.32000/sentencepiece.model'. But currently only gs://scifive is accessible.

Could you please release the vocab or share with us how exactly did you obtain the sentencepiece vocab so that we can reproduce the results? Thank you!

[justinphan3110]

Hey @cnut1648 , Sorry for the late response. On the Chemprot task, we found out that we used the same eval approach as in scibert. Therefore, Chemprot result reported in the paper is not comparable to other recent works which use different eval approaches.

The new results for Scifive on chemprot should be: ~78

Let me know if you still want the code to reproduce this result

[cnut1648]

Hi @justinphan3110 thank you for the info!
It would still be nice to reproduce it. I am working on an experiment on low-resource medical RE setting so I probably will reproduce your results and also extend to low-resource chemprot & ddi.
If it's not a big burden to ask, may we access to relevant training files (eg vocab) to reproduce chemprot and ddi experiments?
Thank you!

[justinphan3110]

@cnut1648 I have just updated the script to use the similar chemprot dataset as the BLURB Leaderboard. You can access it here. Let me know if you still have any question

[cnut1648]

Hi @justinphan3110 thanks! I can reproduce the results now! Great work!