-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
provide data files? #4
Comments
We are working on resolving this in a more universal way shortly. Atm, check out the files and directions here. Note that the files that contain all the attentions and the hidden representations for an entire corpus can be very large. Make sure you have at least 10GB of space before performing this operation on the wizard of oz txt |
I see, thanks for the explanation. There seems to be some issue with tokenization (I didn't read the code in-depth) when I tried to generate the attention files. I guess I will wait for a more universal solution. At this point, is it possible to run the demo on single sentences without caching a large corpus? In theory that's doable right? |
The most common issue with tokenization is an environment one -- make sure you run So long as there are no "file not found" issues when looking for the hdf5 and faiss files on starting the server, you should be able to run just to play with the attention graph. Without those, you will not be able to search across the inner representations, however. |
Hi, thanks for the corpus creating scripts. They're very helpful. I got an index error with BPE/spacy tokenization as shown below:
In IndexError designed to raise under certain circumstances? If not how can I solve it? Thank you very much. |
This looks like a separate bug for the spaCy - BPE alignment. Will look into it in a separate issue. Looking at the two tokenized lists, it looks like the contraction |
Hi,
Is it possible that you provide some example data files? If I run the backend, it prompts errors such as
exbert/server/data/woz/embeddings/combined.hdf5', errno = 2, error message = 'No such file or directory'
. I tried to use the data tools to create such embeddings, but I encountered multiple errors during that process as well.Thank you.
The text was updated successfully, but these errors were encountered: