Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

XLA: random numbers are the same across session.run calls #6854

Closed
yaroslavvb opened this issue Jan 14, 2017 · 9 comments
Closed

XLA: random numbers are the same across session.run calls #6854

yaroslavvb opened this issue Jan 14, 2017 · 9 comments
Assignees
@hawkinsp
Labels
type:bug Bug

Comments

@yaroslavvb
Copy link
Contributor

Not sure if that's intended, but that changes behavior of training pipelines:

from tensorflow.contrib.compiler import jit
jit_scope = jit.experimental_jit_scope
with jit_scope(compile_ops=True):
    x = tf.random_uniform(())
    
sess = tf.Session()
print(sess.run(x))
print(sess.run(x))

Output:

0.768917
0.768917
@yaroslavvb yaroslavvb assigned ghost Jan 14, 2017
@yaroslavvb
Copy link
Contributor Author

@Leary for comment if that's normal

@hawkinsp
Copy link
Contributor

That's definitely not working as intended. We don't seem to be setting the RNG seed correctly, which is why you get the same result.

I'll check in a change disabling JIT compilation of the RNG ops until they are fixed.

@hawkinsp
Copy link
Contributor

Actually, it turns out that Tensorflow is constant-folding away the _XlaLaunch ops containing the random-number generation code. Fix coming shortly (mark _XlaLaunch as stateful).

@yaroslavvb
Copy link
Contributor Author

Good to know it's an easy fix. BTW, if you setup the internal github/google email it'll add you to TensorFlow org automatically and I'll be able to assign future XLA issues to you (martin knows the place for the mapping)

@yaroslavvb yaroslavvb unassigned ghost Jan 15, 2017
@Leary
Copy link

Leary commented Jan 15, 2017 via email

@hawkinsp
Copy link
Contributor

hawkinsp commented Jan 15, 2017 •
edited

Ah, I see, I can't be assigned. I emailed Martin to add me to the Github org.

@vrv
Copy link

vrv commented Jan 15, 2017

@hawkinsp invited

@vrv
Copy link

vrv commented Jan 15, 2017

Also it is @leary-g I think

@jart jart added the type:bug Bug label Jan 16, 2017
@hawkinsp hawkinsp self-assigned this Jan 16, 2017
caisq pushed a commit to caisq/tensorflow that referenced this issue Jan 19, 2017
@hawkinsp @tensorflower-gardener
[TF:XLA] Mark _XlaLaunch as a stateful op since XLA's random-number g…
88499af 
…enerators are stateful.

Prevents Tensorflow from constant-folding _XlaLaunch ops containing random-number ops. Fixes Github issue: tensorflow#6854

Enable RandomStandardNormal for the XLA CPU backend.
Change: 144716342
@yaroslavvb
Copy link
Contributor Author

yaroslavvb commented Jan 21, 2017 •
edited

@hawkinsp btw, if you include string like below in CL public description, it'll automatically close the issue when commit is merged:
Fixes issue: #6854
("Fixes Github issue: #6854" does not trigger it)

benoitsteiner pushed a commit to benoitsteiner/tensorflow that referenced this issue Jan 27, 2017
@hawkinsp @benoitsteiner
[TF:XLA] Mark _XlaLaunch as a stateful op since XLA's random-number g…
5f90cb8 
…enerators are stateful.

Prevents Tensorflow from constant-folding _XlaLaunch ops containing random-number ops. Fixes Github issue: tensorflow#6854

Enable RandomStandardNormal for the XLA CPU backend.
Change: 144716342
gunan pushed a commit that referenced this issue Feb 7, 2017
@hawkinsp @gunan
[TF:XLA] Mark _XlaLaunch as a stateful op since XLA's random-number g…
bd371a3 
…enerators are stateful.

Prevents Tensorflow from constant-folding _XlaLaunch ops containing random-number ops. Fixes Github issue: #6854

Enable RandomStandardNormal for the XLA CPU backend.
Change: 144716342
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@hawkinsp hawkinsp

Labels
type:bug Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@yaroslavvb @jart @hawkinsp @vrv @Leary