RuntimeError: torch.cat(): expected a non-empty list of Tensors

[G-force78]

After running the demo notebook specifying lineart_realistic on a custom video I get this error

RuntimeError Traceback (most recent call last)

in <cell line: 1>()
----> 1 res = run(input_ns)
2 save_dir_name = 'animation.16'
3 save_dir = f'assets/notebook-generated/{save_dir_name}'
4 os.makedirs(save_dir, exist_ok=True)
5 if len(res) == 3:

3 frames

/content/RAVE/pipelines/sd_controlnet_rave.py in process_image_batch(self, image_pil_list)
313 control_torch_list.append(control_image)
314 image_torch_list.append(ipu.pil_img_to_torch_tensor(image_pil))
--> 315 control_torch = torch.cat(control_torch_list, dim=0).to(self.device)
316 img_torch = torch.cat(image_torch_list, dim=0).to(self.device)
317 torch.save(control_torch, os.path.join(self.controls_path, 'control.pt'))

RuntimeError: torch.cat(): expected a non-empty list of Tensors

[G-force78]

So I tried the same with the demo video truck and got the same error

content/data/mp4_videos/truck.mp4
Frame count: 0
{ 'batch_size': 2,
'batch_size_vae': 1,
'cond_step_start': 0.0,
'control_path': '/content/generated/data/controls/truck/lineart_realistic_3x3_1',
'controlnet_conditioning_scale': 1.0,
'controlnet_guidance_end': 1.0,
'controlnet_guidance_start': 0.0,
'give_control_inversion': True,
'grid_size': 3,
'guidance_scale': 7.5,
'hf_cn_path': 'lllyasviel/control_v11p_sd15_lineart',
'hf_path': 'runwayml/stable-diffusion-v1-5',
'image_pil_list': [],
'inverse_path': '/content/generated/data/inverses/truck/lineart_realistic_None_3x3_1',
'inversion_prompt': '',
'is_ddim_inversion': True,
'is_shuffle': True,
'model_id': 'None',
'negative_prompts': '',
'num_inference_steps': 25,
'num_inversion_step': 25,
'pad': 1,
'positive_prompts': 'insects crawling around a track',
'preprocess_name': 'lineart_realistic',
'sample_size': 0,
'save_folder': 'truck',
'save_path': '/content/results/12-21-2023/truck/insects crawling around a '
'track-00001',
'seed': 0,
'video_name': 'truck',
'video_path': '/content/data/mp4_videos/truck.mp4'}

---

RuntimeError Traceback (most recent call last)

in <cell line: 1>()
----> 1 res = run(input_ns)
2 save_dir_name = 'truck'
3 save_dir = f'assets/notebook-generated/{save_dir_name}'
4 os.makedirs(save_dir, exist_ok=True)
5 if len(res) == 3:

3 frames

/content/RAVE/pipelines/sd_controlnet_rave.py in process_image_batch(self, image_pil_list)
313 control_torch_list.append(control_image)
314 image_torch_list.append(ipu.pil_img_to_torch_tensor(image_pil))
--> 315 control_torch = torch.cat(control_torch_list, dim=0).to(self.device)
316 img_torch = torch.cat(image_torch_list, dim=0).to(self.device)
317 torch.save(control_torch, os.path.join(self.controls_path, 'control.pt'))

RuntimeError: torch.cat(): expected a non-empty list of Tensors

[ozgurkara99]

Hello, it seems that the sample_size is assigned as 0, meaning that either the video is in the wrong path, or for some reason it cannoy read the video. Also note that the frame amount should be more than 9 since you are using 3 as the grid size, meaning that 3x3 grid is going to be initialized. Thanks

[G-force78]

Hello, it seems that the sample_size is assigned as 0, meaning that either the video is in the wrong path, or for some reason it cannoy read the video. Also note that the frame amount should be more than 9 since you are using 3 as the grid size, meaning that 3x3 grid is going to be initialized. Thanks

The video 'truck' is in the path designated by the repo /content/RAVE/data/mp4_videos , there does not seem to be anywhere to change the frame amount? Should the video be preprocessed, such as broken down into frames first? If so the notebook does not have a block to do that.

I tried another way using webui script and it did start processing it but ran out of memory, video was 25fps for one second, 512x512