Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

questions with places512 trainning #48

Open
YIFanH opened this issue Nov 14, 2022 · 8 comments
Open

questions with places512 trainning #48

YIFanH opened this issue Nov 14, 2022 · 8 comments

Comments

@YIFanH
Copy link

YIFanH commented Nov 14, 2022

HI @fenglinglwb ,thanks for your work, i use source code to train places2 datasets, but meet the self._load_raw_image out of bounds in datasets/dataset_512.py, but executing this file alone is no problem. thanks for your reply!

@fenglinglwb
Copy link
Owner

Please provide more error information.

@YIFanH
Copy link
Author

YIFanH commented Nov 14, 2022

In datasets/dataset_512.py line 218,raw_idx > total files number in trainning,for example , 992 imgs in test forder, IndexError: index 992 is out of bounds for axis 0 with size 992.
def _load_raw_image(self, raw_idx):
fname = self._image_fnames[raw_idx]
with self._open_file(fname) as f:
if pyspng is not None and self._file_ext(fname) == '.png':
image = pyspng.load(f.read())
else:
image = np.array(PIL.Image.open(f))
if image.ndim == 2:
image = image[:, :, np.newaxis] # HW => HWC

@fenglinglwb
Copy link
Owner

fenglinglwb commented Nov 14, 2022

Please check the metric you used. For example, when adopting 'fid36k5', we need to prepare 36.5k images. Just select a proper metric.

@YIFanH
Copy link
Author

YIFanH commented Nov 14, 2022

Please check the metric you used. For example, when adopting 'fid36k5', we need to prepare 36.5k images. Just select a proper metric.
thanks for your reply , I update fid_places in metrics/metric_main.py like this:
def fid_places(opts):
opts.dataset_kwargs.update(max_size=None, xflip=False)
# fid = frechet_inception_distance.compute_fid(opts, max_real=36500, num_gen=36500)
fid = frechet_inception_distance.compute_fid(opts, max_real=900, num_gen=900)
return dict(fid36k5_full=fid)
and run with --metrics fid_places.

but I meet other cuda memory problems,batchsize irrelevance.

@fenglinglwb
Copy link
Owner

Which error?

@YIFanH
Copy link
Author

YIFanH commented Nov 14, 2022

in trainning cuda memory out with one V100 gpu batch with 8,i guess somthing wrong with cuda mem is not cleared.

@fenglinglwb
Copy link
Owner

This needs to be solved by yourself

@YIFanH
Copy link
Author

YIFanH commented Nov 14, 2022

yes , i need read code more, thanks for your work and reply

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants