Bus error (core dumped)

[Reid00]

DGLBACKEND=pytorch dglke_train --model_name TransE_l2 --dataset patient --batch_size 1000 --neg_sample_size 200 --hidden_dim 400 --gamma 19.9 --lr 0.25 --max_step 24000 --log_interval 100 --batch_size_eval 16 -adv --regularization_coef 1.00E-09 --test --gpu 0 1 2 3 --mix_cpu_gpu --data_path ./data/ --format raw_udd_hrt --data_files train.txt valid.txt test.txt --neg_sample_size_eval 10000

does this error means out of memory?

[classicsong]

Which DGL version you are using?

[Reid00]

it's 0.4.3 follow the official guideline https://dglke.dgl.ai/doc/install.html

[classicsong]

Can you run dmesg to see the linux error log?
Further which pytorch version are you using?

[Reid00]

thank you. i run dmesg , result as below and pytorch version is 1.7.0 + cuda 10.2

[classicsong]

Can you do some simple math to see if you have enough memory to hold the embeddings?
It at least needs: 4 * hidden_dim * number_of_nodes + 16 * number_of_edges to hold the whole graph for training without considering the intermediate memory requirements.

[Reid00]

i think memory is enough.
i'm not sure why this happens. when i use single GPU is run well
DGLBACKEND=pytorch dglke_train --model_name TransE_l2 --dataset patient --batch_size 1000 --neg_sample_size 200 --hidden_dim 400 --gamma 19.9 --lr 0.25 --max_step 24000 --log_interval 100 --batch_size_eval 16 -adv --regularization_coef 1.00E-09 --test --gpu 0 --mix_cpu_gpu --data_path ./data/ --format raw_udd_hrt --data_files train.txt valid.txt test.txt --neg_sample_size_eval 10000
go well

DGLBACKEND=pytorch dglke_train --model_name TransE_l2 --dataset patient --batch_size 1000 --neg_sample_size 200 --hidden_dim 400 --gamma 19.9 --lr 0.25 --max_step 24000 --log_interval 100 --batch_size_eval 16 -adv --regularization_coef 1.00E-09 --test --gpu 0 1 2 3 --mix_cpu_gpu --data_path ./data/ --format raw_udd_hrt --data_files train.txt valid.txt test.txt --neg_sample_size_eval 10000
error with Buss error

[classicsong]

Then, it maybe related to multi-gpu implementation. Did you try pytorch 1.6?
What kind of hardware you are using?

[Reid00]

i will try pytorch 1.6 later

[classicsong]

Thank you.

[Reid00]

i tried torch 1.6 cuda 10.2 and python 3.8. it's still the same error. so what other reason maybe cause to this issue?

[classicsong]

Can you try run it with GDB and use backtrace to see where cause the crash?

[VoVAllen]

Bus error usually means your shared memory size is not big enough. If you are using docker, please pass --shm-size="4G" or bigger value when running the container.