Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

节点多的话跑不通，吃内存 #18

Open
delphi1987 opened this issue Jun 12, 2019 · 5 comments
Open

节点多的话跑不通，吃内存 #18

delphi1987 opened this issue Jun 12, 2019 · 5 comments

Comments

@delphi1987
Copy link

48万个节点，32G内存跑不同，在_create_A_L上报内存不够。
_create_A_L种构造矩阵的方式是否可优化，稀疏矩阵这样存储太浪费。
@shenweichen
Copy link
Owner

可以使用numpy或scipy中的sparse matrix来存储

@wqw123
Copy link

wqw123 commented Jun 22, 2019

48万个节点，32G内存跑不同，在_create_A_L上报内存不够。
_create_A_L种构造矩阵的方式是否可优化，稀疏矩阵这样存储太浪费。

您好，请问该问题解决了吗？如何解决的

@MarkovSc
Copy link

MarkovSc commented Aug 5, 2019

可以使用numpy或scipy中的sparse matrix来存储
但是在训练的时候，还得用array 进行input，48万个节点，目测100G都不够用。

@shenweichen
Copy link
Owner

@MarkovSc 改成Sparse placeholder输入

@anmingyu11
Copy link

这个并行实现是不是有点问题？不管开多少worker都不是并行，为什么不用py内置的并行模块呢？

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@MarkovSc @shenweichen @anmingyu11 @wqw123 @delphi1987