Memory Issue when running baseline models

[cenyk1230]

Hi, Thanks for the great challenge.

I tried to reproduce the "concat" baseline using

CUDA_VISIBLE_DEVICES=0,1,2,3 dglke_train --model_name ComplEx \
--hidden_dim 100 --gamma 3 --lr 0.1 --regularization_coef 1e-9 \
--valid --test -adv --mix_cpu_gpu --num_proc 4 --num_thread 4 \
--gpu 0 1 2 3 \
--async_update --force_sync_interval 50000 --no_save_emb \
--print_on_screen --encoder_model_name concat -de -dr --save_path saved

but it reported MemoryError.
What's the memory requirement of running the "concat" baseline?

Hardware:

• 256G Memory
• 8 * GeForce RTX 2080 Ti (11G)

Environments

• python 3.7.10
• torch 1.7.0
• ogb 1.3.1
• dgl 0.4.3.post2
• cuda 10.1

The entire output is

Using backend: pytorch                                                                                                          
To build training dataset                                                                                                       
|Train|: 501160482                                                                                                              
random partition 501160482 edges into 4 parts                                                                                   
part 0 has 125290121 edges                                                                                                      
part 1 has 125290121 edges                                                                                                      
part 2 has 125290121 edges                                                                                                      
part 3 has 125290119 edges                                                                                                      
/home/yukuo/anaconda3/envs/py3.7/lib/python3.7/site-packages/dgl/base.py:25: UserWarning: multigraph will be deprecated.DGL will
 treat all graphs as multigraph in the future.                                                                                  
  warnings.warn(msg, warn_type)                                                                                                 
Training dataset built, it takes 1115 seconds                                                                                   
Building training sampler for proc 0                                                                                            
Training sampler for proc 0 created, it takes 18.501158714294434 seconds                                                        
Building training sampler for proc 1                                                                                            
Training sampler for proc 1 created, it takes 19.520564794540405 seconds                                                        
Building training sampler for proc 2                                                                                            
Training sampler for proc 2 created, it takes 20.290805101394653 seconds                                                        
Building training sampler for proc 3                                                                                            
Training sampler for proc 3 created, it takes 20.442392349243164 seconds                                                        
To create eval_dataset                                                                                                          
eval_dataset created, it takes 313 seconds                                                                                      
creating valid sampler for proc 0                                                                                               
0 425146                                                                                                                        
Valid sampler for proc 0 created, it takes 0.002320528030395508 seconds                                                         
creating valid sampler for proc 1                                                                                               
425146 850292                                                                                                                   
Valid sampler for proc 1 created, it takes 4.172325134277344e-05 seconds                                                        
creating valid sampler for proc 2                                                                                               
850292 1275438                                                                                                                  
Valid sampler for proc 2 created, it takes 3.0517578125e-05 seconds                                                             
creating valid sampler for proc 3                                                                                               
1275438 1700584                                                                                                                 
Valid sampler for proc 3 created, it takes 2.47955322265625e-05 seconds                                                         
creating test sampler for proc 0                                                                                                
0 339825                                                                                                                        
Test sampler for proc 0 created, it takes 0.004835367202758789 seconds                                                          
creating test sampler for proc 1                                                                                                
339825 679651                                                                                                                   
Test sampler for proc 1 created, it takes 0.0003917217254638672 seconds                                                         
creating test sampler for proc 2                                                                                                
679651 1019477                                                                                                                  
Test sampler for proc 2 created, it takes 0.0003933906555175781 seconds                                                         
creating test sampler for proc 3                                                                                                
1019477 1359303                                                                                                                 
Test sampler for proc 3 created, it takes 0.0004000663757324219 seconds 
To create model                                                                                                                 
False False                                                                                                                     
Model created, it takes 281.9187514781952 seconds                                                                               
Total initialize time 2186.089 seconds                                                                                          
Traceback (most recent call last):                                                                                              
  File "/home/yukuo/anaconda3/envs/py3.7/bin/dglke_train", line 33, in <module>                                                 
    sys.exit(load_entry_point('dglke', 'console_scripts', 'dglke_train')())                                                     
  File "/home/yukuo/OGB-LSC/wikikg90m/dgl-ke-ogb-lsc/python/dglke/train.py", line 346, in main                                  
    proc.start()                                                                                                                
  File "/home/yukuo/anaconda3/envs/py3.7/lib/python3.7/multiprocessing/process.py", line 112, in start                          
    self._popen = self._Popen(self)                                                                                             
  File "/home/yukuo/anaconda3/envs/py3.7/lib/python3.7/multiprocessing/context.py", line 223, in _Popen                         
    return _default_context.get_context().Process._Popen(process_obj)                                                           
  File "/home/yukuo/anaconda3/envs/py3.7/lib/python3.7/multiprocessing/context.py", line 284, in _Popen                         
    return Popen(process_obj)                                                                                                   
  File "/home/yukuo/anaconda3/envs/py3.7/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__              
    super().__init__(process_obj)                                                                                               
  File "/home/yukuo/anaconda3/envs/py3.7/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__                     
    self._launch(process_obj)                                                                                                   
  File "/home/yukuo/anaconda3/envs/py3.7/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch               
    reduction.dump(process_obj, fp)                                                                                             
  File "/home/yukuo/anaconda3/envs/py3.7/lib/python3.7/multiprocessing/reduction.py", line 60, in dump                          
    ForkingPickler(file, protocol).dump(obj)                                                                                    
MemoryError  

Best Regards,
Yukuo Cen

[cenyk1230]

Hi, I find a machine with 512G CPU memory to run the script. But it still doesn't work.
The error message is

Traceback (most recent call last):
  File "/home/yukuo/anaconda3/envs/py3.7/bin/dglke_train", line 33, in <module>
    sys.exit(load_entry_point('dglke', 'console_scripts', 'dglke_train')())
  File "/home/yukuo/OGB-LSC/wikikg90m/dgl-ke-ogb-lsc/python/dglke/train.py", line 346, in main
    proc.start()
  File "/home/yukuo/anaconda3/envs/py3.7/lib/python3.7/multiprocessing/process.py", line 112, in start
    self._popen = self._Popen(self)
  File "/home/yukuo/anaconda3/envs/py3.7/lib/python3.7/multiprocessing/context.py", line 223, in _Popen
    return _default_context.get_context().Process._Popen(process_obj)
  File "/home/yukuo/anaconda3/envs/py3.7/lib/python3.7/multiprocessing/context.py", line 284, in _Popen
    return Popen(process_obj)
  File "/home/yukuo/anaconda3/envs/py3.7/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/home/yukuo/anaconda3/envs/py3.7/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
    self._launch(process_obj)
  File "/home/yukuo/anaconda3/envs/py3.7/lib/python3.7/multiprocessing/popen_spawn_posix.py", line 47, in _launch
    reduction.dump(process_obj, fp)
  File "/home/yukuo/anaconda3/envs/py3.7/lib/python3.7/multiprocessing/reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'ComplExScore.create_neg.<locals>.fn'

[hyren]

Hi, 512 GB should be enough. Can you run Roberta or shallow?

[cenyk1230]

I think the versions really matter. I finally follow the suggestion from awslabs/dgl-ke#170 (comment) to use python3.6 and torch1.6.0 and it works.

[hyren]

Sounds good. Let me know if you have any further questions.

[shravankumar147]

Are you using any cloud platforms or personal machines? Trying to understand where(on which platform) to get started?