error on running "make learn-vectors-redis"

[llnlln]

Hi @jjhenkel,

A Segmentation fault when I run the make learn-vectors-redis . could you give me some advice?

Thanks!

[jjhenkel]

Hi @llnlln ,

Try getting the latest version of the arbitrary-repos branch and running that command again. I just pushed a fix.

Thanks!

[llnlln]

Hi @jjhenkel

I ran the following command and got the same error. The difference is that the array size is 16320875724 and 510027366, respectively.

git status
git pull 
make learn-vectors-redis

The output is in the output.txt file
output.txt

Thanks!

[jjhenkel]

Hi @llnlln ,

I'm trying to reproduce this on my machine---the patch I pushed did fix things for me but, just to be sure, I'll run it on a fresh clone of the repository and see if the issue re-appears.

In the meantime what system are you running this on (OS/number of cores/amount of RAM)? Could you try completely removing the repository and then re-cloning, checking out the arbitrary-repos branch, and running make end-to-end-redis ; make learn-vectors-redis.

At this point the tool has generated the Abstracted Symbolic Traces---this means that, as a workaround, you could always just run your favorite word vector learner on the generated traces (the make learn-vectors-redis command is just going to run GloVe on the generated traces using default settings).

Thanks!

[jjhenkel]

Follow up on this: I am no longer able to reproduce this. I just cloned a fresh copy of the repo, checked out the arbitrary-repos branch, ran make end-to-end-redis followed by make learn-vectors-redis and got reasonable output at the end:

2019-03-01 03:57:56,651 - glove2word2vec - INFO - Converted model with 6150 vectors and 300 dimensions
[GloVe] Complete!
[GloVe] Vectors saved in '/output/vectors'
[code-vectors] Learner finished. Output saved in /home/jjhenkel/code-vectors-artifact/artifacts/redis
[code-vectors] Running demo using Gensim and our freshly learned vectors...
docker run -it --rm \
	-v /home/jjhenkel/code-vectors-artifact/artifacts:/artifacts \
	--entrypoint python \
	jjhenkel/glove:gz \
	/app/redis-demo.py
[code-vectors] 5 closest words to "setTypeAdd"
[code-vectors]  #0. setTypeSize
[code-vectors]  #1. setTypeRemove
[code-vectors]  #2. setTypeInitIterator
[code-vectors]  #3. createIntsetObject
[code-vectors]  #4. setTypeReleaseIterator

[code-vectors] 5 closest words to "streamIteratorGetID"
[code-vectors]  #0. streamIteratorStart
[code-vectors]  #1. streamIteratorStop
[code-vectors]  #2. ?->lp
[code-vectors]  #3. addReplyStreamID
[code-vectors]  #4. streamIteratorGetID_$EQ_0

[code-vectors] 5 closest words to "clusterAddSlot"
[code-vectors]  #0. clusterDelSlot
[code-vectors]  #1. clusterAddNode
[code-vectors]  #2. countKeysInSlot_$EQ_0
[code-vectors]  #3. !->importing_slots_from[?]
[code-vectors]  #4. countKeysInSlot

[code-vectors] 5 closest words to "clusterLoadConfig"
[code-vectors]  #0. clusterLockConfig
[code-vectors]  #1. clusterLockConfig_$NEQ_-1
[code-vectors]  #2. clusterLoadConfig_$EQ_-1
[code-vectors]  #3. clusterLoadConfig_$NEQ_-1
[code-vectors]  #4. loadAppendOnlyFile

[code-vectors] 5 closest words to "!->importing_slots_from[?]"
[code-vectors]  #0. clusterBumpConfigEpochWithoutConsensus_$NEQ_0
[code-vectors]  #1. !->migrating_slots_to[?]
[code-vectors]  #2. clusterAddSlot
[code-vectors]  #3. sentinelHandleConfiguration_$EQ_0
[code-vectors]  #4. sdssplitargs

[code-vectors] 5 closest words to "$ERR"
[code-vectors]  #0. $RET_EPERM
[code-vectors]  #1. $RET_ENOENT
[code-vectors]  #2. malloc_$EQ_0
[code-vectors]  #3. rdbSaveLen_$EQ_-1
[code-vectors]  #4. redisContextWaitReady_$NEQ_0

[code-vectors] Demo complete.

[llnlln]

Hi @jjhenkel ,

I just cloned a fresh copy of the repo, checked out the arbitrary-repos branch, ran make end-to-end-redis followed by make learn-vectors-redis and got the same error.
I ran this on Linux version 4.15.0-42-generic(Ubuntu 5.4.0-6ubuntu1~16.04.10)/Intel® Core™ i5-4570 CPU @ 3.20GHz × 4 /memory 3.5 GiB.
Could you give me some advice?

Thanks!

[jjhenkel]

Hi @llnlln ,

Sorry for the delay---I've been trying to find a machine I can reproduce this on.

For now my advice to you is to learn vectors using your method of choice without using the version of GloVe I provide as a Docker image. You should have the generated traces on disk (since the previous steps in the pipeline worked) so you can simply run any word-vector learner using those traces as your corpus.

If you need an example of how to run one of these learners you can see the commands I use for learning vectors via GloVe in this file. Other tools like word2vec and fasttext also provide good documentation and examples of their usage.

[llnlln]

Hi @jjhenkel ,

Thank you very much for your reply !