Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory Error while running the training code #50

Open
Rufaida94 opened this issue Jun 21, 2021 · 3 comments
Open

Memory Error while running the training code #50

Rufaida94 opened this issue Jun 21, 2021 · 3 comments

Comments

@Rufaida94
Copy link

Rufaida94 commented Jun 21, 2021

Hi @wuyifan18 , thank you for the great tool. It works perfectly with a very small dataset, but whenever I try running it with a larger dataset I get this error during the training phase:

[enforce fail at CPUAllocator.cpp:64] . DefaultCPUAllocator: can't allocate memory: you tried to allocate 278528000 bytes. Error code 12 (Cannot allocate memory)

My machine has large RAM but I am not sure why is this happening or how can I resolve this issue? Or how can we edit the code so that we use less memory space with each epoch? now each epoch takes approximately 1 GB of memory.

Any suggestion is highly appreciated.

Thanks

@wuyifan18
Copy link
Owner

Hi @Rufaida94, try reducing the batch_size?

@Rufaida94
Copy link
Author

I've reduced the batch size and removed any writing into memory except for the model and it worked fine. Thanks

@Rufaida94
Copy link
Author

Rufaida94 commented Jun 22, 2021

@wuyifan18 Is there any way to make the code run faster (especially the training code) for a very large dataset? Currently, it is taking about 1 hour for each epoch with either a GPU or a CPU.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants