Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Random in nntrainer #167

Closed
kparichay opened this issue Jun 15, 2020 · 5 comments · Fixed by #179
Closed

Random in nntrainer #167

kparichay opened this issue Jun 15, 2020 · 5 comments · Fixed by #179

Comments

@kparichay
Copy link
Member

kparichay commented Jun 15, 2020

random() returns random value in nntrainer.
This results in the indeterministic training of the models in the unit tests.

How about fixing the seed in testing so that we can ensure that the training result stays the same with newer changes.

@jijoongmoon please advise if there should be an API function for this?

@taos-ci
Copy link
Collaborator

taos-ci commented Jun 15, 2020

:octocat: cibot: Thank you for posting issue #167. The person in charge will reply soon.

@zhoonit
Copy link
Contributor

zhoonit commented Jun 15, 2020

Setting seeds seems great idea to me (also related to #133 )
Could you check if setting seeds would make a reproducible result as well?
I am concerning that since databuffer is designed to run concurrently, calling order of random() is not deterministic.

@kparichay
Copy link
Member Author

@zhoonit Yes, #133 is good and is definitely needed.
Thanks for your input, I was not very familiar with databuffer, I will check the corresponding files and then update this issue.

@jijoongmoon
Copy link
Collaborator

jijoongmoon commented Jun 15, 2020

That's good idea and it will make nntrainer more deterministic. I tried to do this in other ways like #148 which has python code to generate the input, output and golden data. It is used for testing conv2d and pooling layer for forwarding and compared with output from nntrainer. (Tensorflow is used to generate the golden data). However, we do not have tensorflow for tizen so that tar file is used.

@kparichay
Copy link
Member Author

Current major issue is initialization. I will for now make the seed fixed.
Later, maybe add an interface for user to set it.

kparichay added a commit to kparichay/nntrainer that referenced this issue Jun 16, 2020
Adding determinism to the random number generators in the library
DataBuffer has multiple threads but single thread of train/valid/test
which run in sequence in my understanding

Resolves nnstreamer#167

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>
kparichay added a commit to kparichay/nntrainer that referenced this issue Jun 16, 2020
Adding determinism to the random number generators in the library
DataBuffer has multiple threads but single thread of train/valid/test
which run in sequence in my understanding

Resolves nnstreamer#167

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>
kparichay added a commit to kparichay/nntrainer that referenced this issue Jun 16, 2020
Adding determinism to the random number generators in the library
DataBuffer has multiple threads but single thread of train/valid/test
which run in sequence in my understanding

Resolves nnstreamer#167

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>
kparichay added a commit to kparichay/nntrainer that referenced this issue Jun 16, 2020
Adding determinism to the random number generators in the library
DataBuffer has multiple threads but single thread of train/valid/test
which run in sequence in my understanding

Resolves nnstreamer#167

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>
kparichay added a commit to kparichay/nntrainer that referenced this issue Jun 16, 2020
Adding determinism to the random number generators in the library
DataBuffer has multiple threads but single thread of train/valid/test
which run in sequence in my understanding

Resolves nnstreamer#167

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>
jijoongmoon pushed a commit that referenced this issue Jun 17, 2020
Adding determinism to the random number generators in the library
DataBuffer has multiple threads but single thread of train/valid/test
which run in sequence in my understanding

Resolves #167

Signed-off-by: Parichay Kapoor <pk.kapoor@samsung.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants