You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Could you add maxtasksperchild?
Though it may cause performance issue if new process is created for every iterator, one process per 10 or 20 iteration will not cause performance issue (if it does, it is so fast that it is not needed to prefetch) and it is enough to reduce memory.
The text was updated successfully, but these errors were encountered:
kmaehashi
added
pr-ongoing
Pull-request is being worked on to address the labeled ticket.
and removed
prio:high
High priority. Urgent and needs to be worked on as soon as possible.
labels
Sep 10, 2018
The amount of memory is growing when we use
MultiprocessIterator
in large dataset training. This can be prevented by settingmaxtasksperchild
in usingmultiprocess.Pool
https://github.com/chainer/chainer/blob/master/chainer/iterators/multiprocess_iterator.py#L345
Could you add
maxtasksperchild
?Though it may cause performance issue if new process is created for every iterator, one process per 10 or 20 iteration will not cause performance issue (if it does, it is so fast that it is not needed to prefetch) and it is enough to reduce memory.
The text was updated successfully, but these errors were encountered: