terminate called after throwing an instance of 'std::bad_array_new_length'

[wfang]

Hi sbeamer:

I tried to compile and run the gapbs benchmark. I get the error of std::bad_array_new_length

The steps to reproduce:

1. I run make bench-graphs to download the Graph.

ls benchmark/graphs/raw/ -l
total 31882424
lrwxrwxrwx 1 14 Sep 6 10:37 twitter.el -> twitter_rv.net
-rw-r----- 1 26172210176 Sep 5 18:27 twitter_rv.net
-rw-r----- 1 6475352982 Feb 17 2010 twitter_rv.tar.gz

2. Then I run make bench-run, and get the following error:

./converter -f benchmark/graphs/raw/twitter.el -b benchmark/graphs/twitter.sg
Read Time:           57.28654
terminate called after throwing an instance of 'std::bad_array_new_length'
  what():  std::bad_array_new_length
make: *** [benchmark/graphs/twitter.sg] Aborted

3. I am able to trace and the error coming at this place:

2  0x0000000000408850 in BuilderBase<long, long, long, true>::MakeGraph() () at src/builder.h:235
#3  0x0000000000403560 in main () at src/converter.cc:26
(gdb) n
53	    while (in >> u >> v) {
(gdb) l
48	  }
49	
50	  EdgeList ReadInEL(std::ifstream &in) {
51	    EdgeList el;
52	    NodeID_ u, v;
53	    while (in >> u >> v) {
**54	      el.push_back(Edge(u, v))**;
55	    }
56	    return el;
57	  }

[wfang]

Just realise that the error is due to my change the type size to 64 bit in the benchmark.h file.
typedef int64_t NodeID;
typedef int64_t WeightT;
If I change it back to 32 bit, the error goes away.
typedef int32_t NodeID;
typedef int32_t WeightT;

Not sure why it doesn't work for 64 bit though.

[sbeamer]

Thank you for reporting this!

It would be great to improve support for 64-bit IDs. How much memory does your system have? Running out of memory seems like the most probable cause (http://en.cppreference.com/w/cpp/memory/new/bad_array_new_length).

[wfang]

Hi sbeamer,
Thanks for the reply.
The machine has 1TB memory. I also tested it in another 3TB machine, it failed too for the 64bit.

[sbeamer]

I just built converter with NodeID set to int64_t, and I was unable to reproduce this error when converting twitter. My console output:

Read Time:           222.30987
Build Time:          13.85658
Graph has 61578415 nodes and 1468364884 directed edges for degree: 23
serialized graphs only allowed for 32b IDs

The program will (correctly) exit and not write out the .sg file since the binary encoding expects the 32-bit word width, but it will read the edge list in and build the graph before the exception.

Looking at the code more, I'm not sure if I could imagine how the error could occur within ReadInEL. Your console output prints the read time, which implies ReadInEL completed. Could you double check with gdb where the error occurs. If you aren't running out of memory, perhaps the code is somehow asking for a pvector with a negative size, but I am not sure how expanding to int64_t could cause that.

Did you change anything other than NodeID and WeightT?

[wfang]

Nothing was changed expect NodeID and WeightT.

I think I found out the reason for the error: in my file there is an graph vertex of 4099117456045389565.

grep "4099117456045389565" benchmark/graphs/raw/twitter_rv.net 
12819	4099117456045389565

This will cause issue with FindMaxNodeID in builder.h . The max_seen value will be 4099117456045389565, causing my error.

[sbeamer]

Thank you for debugging this further!

I don't seem to have that line in the twitter_rv.net I downloaded from KAIST. I re-downloaded it last week trying to re-create this issue, and that file too appears to not contain it. Can you compare the MD5 hashes for your files with the ones they provide? I would check both the .tar.gz and the .net. My files do match the MD5 hashes.

If somehow your hashes do match, can you re-run grep with -n and let me know the line number for that crazy line? Thanks!

[sbeamer]

@wfang Did you get a chance to check on this?

[michaelsutton]

Don't know if it's related, but I had trouble downloading the twitter graph correctly as well.
I ended up using curl instead of wget, and that fixed all the issues. Also, the download became blazing fast (minutes rather than hours)
I used the following line to download:
curl 'https://an.kaist.ac.kr/~haewoon/release/twitter_social_graph/twitter_rv.tar.gz' -H 'Connection: keep-alive' --compressed -o twitter_rv.tar.gz

[sbeamer]

Thanks for the suggestion @michaelsutton! On my system, curl (with your arguments) still took hours. I'd be happy to merge a PR for speeding up the Twitter download, since that the slow speed is a common complaint.

[wfang]

Hi sbeamer,

Sorry for the delay in reply.
I revisited the issue and found out it is caused by the downloading issue. I downloaded again with michaelsutton's suggestion and the new file works fine now.

Both the two files are having the same size: 6475352982.
The md5sum for the new file is:
c31b4c2d6f3ae325e516e78b499c46f8 twitter_rv.tar.gz
The oldfile I downloaded before is:
88a1991d2a24ad50c99062864880f89f ./twitter_rv.tar.gz

I think this is un-related to the software itself.
Thanks a lot.