Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to set global random seed? #262

Open
lfwin opened this issue Aug 9, 2016 · 3 comments
Open

How to set global random seed? #262

lfwin opened this issue Aug 9, 2016 · 3 comments

Comments

@lfwin
Copy link

lfwin commented Aug 9, 2016

I try to solve this bug #180, but it would be more convenient if we can make random generator stable, and i used method in http://stackoverflow.com/questions/36288235/how-to-get-stable-results-with-tensorflow-setting-random-seed and changed chainer.py, but this does not work! Does tflearn has some function to set a globle seed for random generator?

edit:I locate the bugs in https://github.com/tflearn/tflearn/issues/180, because image read by PIL maybe size (299, 299) but right one should be (299, 299, 3)

@aymericdamien
Copy link
Member

Have you try to use tflearn.init_graph(seed=...)?
About the other link, is it an error with PIL? Are your images grayscale? maybe reshaping them should works (you will need to edit imagePreloader source).

@lfwin
Copy link
Author

lfwin commented Aug 11, 2016

Hi,
1:
I change code based on residual_network_cifar_10.py, but i believe init_graph(seed=...) function not used when running residual_network_cifar_10.py, because i wanted to change gpu memory fraction and modified Trainer class init function as follows
` self.incr_global_step = tf.assign(self.global_step,
tf.add(self.global_step, 1))

        config = None
        tflearn_conf = tf.get_collection(tf.GraphKeys.GRAPH_CONFIG)

        if tflearn_conf == []:
            config = init_graph(gpu_memory_fraction=0.65)
        if tflearn_conf:
            config = tflearn_conf[0]`

and this actually change gpu memory fraction.
In the same way, I set seed config = init_graph(seed=1, gpu_memory_fraction=0.65) It not work, because loss changes every time I run residual_network_cifar_10.py.

2:
After read img, i add these
if img.mode == 'L': img = convert_color(img, 'RGB')
to check is grayscale and transform into RGB and right now can work except loss is nan.
`class ImagePreloader(Preloader):
def init(self, array, image_shape, normalize=True, grayscale=False):
fn = lambda x: self.preload(x, image_shape, normalize, grayscale)
super(ImagePreloader, self).init(array, fn)

def preload(self, path, image_shape, normalize=True, grayscale=False):
    img = load_image(path)
    if img.mode == 'L':
        img = convert_color(img, 'RGB')
    width, height = img.size

    if width != image_shape[0] or height != image_shape[1]:
        img = resize_image(img, image_shape[0], image_shape[1])
    if grayscale:
        img = convert_color(img, 'L')

    img = pil_to_nparray(img)

    if normalize:
        img /= 255.
    return img`

3
After above modification, the loss is nan, how should i debug? code is based on residual_network_cifar_10.py except input use preloader and some image size relevant params
Edit:Solved by reduce image size from (299, 299) to (220, 200) and batch size is 16, my GPU memory is 12G, maybe memory is too small or some other reasons?

@silgon
Copy link

silgon commented Jan 7, 2018

I'm also having the same issue. Any solutions yet?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants