Accessing the ensembled vectors

[meghmehta]

after running "ninja", where can I find the ensemble vectors?

[rspeer]

In this build process, the labels will appear in build-data/combo840.standardized.conceptnet5.labels and the vectors in build-data/combo840.l1.standardized.conceptnet5.npy. The filenames are elaborate because this repository builds many different variations of the system to compare in an evaluation.

There is also a new build process that's part of the conceptnet5 repository, which will give you more up-to-date vectors in data/vectors/numberbatch.h5, as well as the tools you need to look up ConceptNet nodes in that vector space.

[meghmehta]

Thank you - my problem is that I can't find any build-data folder in the repository and so I am not sure where to find these files... would you be able to guid me in this?

[rspeer]

That's where the files created by ninja go.

What happened when you ran ninja in the code/ directory? Did it crash?

[meghmehta]

so I've been running the 16.04 branch....it basically runs through build_conceptnet_retrofitting() which is called in ninja.py. But there's only a source_data folder, I can't find a build_data folder...

[rspeer]

Oh. You haven't run the build yet.

Running ninja.py outputs a file called build.ninja, which contains instructions to the Ninja build system (https://ninja-build.org/) for how to build ConceptNet Numberbatch. After that, you run ninja to actually run the build.

[meghmehta]

Yes I did run ninja as well... like i ran ninja.py and then I ran ninja but I don't get any new outputs...

[meghmehta]

I actually get the error:
Importerror: cannot import ninja from ninja

[rspeer]

Can you copy and paste the actual output, please?

[rspeer]

What's particularly peculiar about saying you got "Importerror: cannot import ninja from ninja" is that it looks almost like a Python error, and this makes me doubt you're actually running Ninja, which is written in C++.

[meghmehta]

I actually don't have that error anymore...when I run ninja it gives the following output:
log: warning: no configuration file specified, using default values
log: ninja version 0.1.3 initializing
log: magic group: gid=0 (wheel)
log: entering main loop
log: generating initial pid array..
log: now monitoring process activity

How long would it take for it to complete? It's been running for > 1 hour now.

[rspeer]

It should take a day or so. Keep in mind that this process runs all the conditions of the experiment.

[meghmehta]

okay thank you!

[meghmehta]

Is it GPU enabled?

[rspeer]

Man, how difficult to set up do you want it to be?

This task is memory-constrained, so I think it would actually run much worse on a GPU.