KeyError: "['file_path_mask_eroded_3'] not in index" #119
Comments
|
@XYAskWhy Hi, did you generate metadata? If you have already done that then open your metadata csv and check which columns are available. Remember that you can choose how to generate your target masks so your csv may contain different columns. You can choose which column should be used as target masks in Y_COLUMNS = ['file_path_mask_eroded_0_dilated_0'] |
|
I also encountered this problem. l will have a try. Thanks! |
|
Thanks @jakubczakon , I had done |
|
@XYAskWhy After I executed |
|
@dslwz2008 If you prepare metadata first, you need to redo it after you prepare mask. Then the newly generated csv file will include a extra column like |
|
@XYAskWhy @dslwz2008 I will fix the readme today but yes as @XYAskWhy when metadata is created it looks for the folders with target masks and creates the columns based on that information.
I hope this helps! |
|
I re-executed commands |
|
@dslwz2008 what are your paths in the data_dir: /path/to/data
meta_dir: /path/to/data
masks_overlayed_dir: /path/to/masks_overlayed
masks_overlayed_eroded_dir: /path/to/masks_overlayed_eroded
experiment_dir: /path/to/work/dir
Can you confirm that your masks did generate? The mask overlayed folder should be around 100G |
|
This is my neptune.yaml:
I am not sure if the |
|
Ok I see. You just need to have something like:
and it will create this particular setting with eroded2_dilated_3 automatically. images_path_to_write = images_path
masks_overlayed_dir_ = masks_overlayed_dir[:-1]
masks_dir_prefix = os.path.split(masks_overlayed_dir_)[1]
masks_overlayed_sufix_to_write = []
for masks_dir in os.listdir(meta_dir):
masks_dir_name = os.path.split(masks_dir)[1]
if masks_dir_name.startswith(masks_dir_prefix):
masks_overlayed_sufix_to_write.append(masks_dir_name[len(masks_dir_prefix):])So you need to define your path ending with |
|
@dslwz2008 @XYAskWhy by the way I updated the readme. |
|
@dslwz2008 also I would change the
to something particular to this experiment. All the models will be saved in that directory so I am not sure if you want to have it as generic as open-solution-mapping-challenge.
or something like that. |
|
OK. Thanks! |
|
Well you misspecified the masks_overlayed_dir: So I dont think you have that folder. Now there are 2 options. You could either specify it correctly: masks_overlayed_dir: /home/shenshen/Programs/masks_overlayed/and rerun generation of the masks (takes time) or you could simply go mv /home/shenshen/Programs/mc_dat_eroded_2_dilated_3 /home/shenshen/Programs/masks_overlayedand rerun metadata creation |
|
Thank you very much! I understand. I did not create masks_overlayed folder before prepare_masks. |
|
this one should actually be dropped It is a remnant of older days when we only thought of 2 configurations of those target masks :) I will drop it from readme and yamls |
|
well you generated all those masks with prepare_masks you just put it in a wrong directory |
|
I finally configured correctly！ Unfortunately, I did not find the column file_path_mask_eroded_0_dilated_0 in the generated stage1_metadata.csv file... |
|
Well metadata generation should be pretty fast it's only filepath munging. Also since you are generating masks with erosion 2 and dilation 3 your path is actually |
|
This is the head of stage1_metadata.csv. No similar column appears. |
|
Did you change the mask_overlayed dir in neptune.yaml to masks_overlayed_dir: /home/shenshen/Programs/masks_overlayed/and recreated the metadata ? |
|
Yes, I have changed the masks_overlayed_dir in the neptune.yaml and delete stage1_metadata.csv. Then I ran the prepare_metadata again. The result is the same as in the picture above. |
|
what does this folder |
|
|
|
Okey, I checked on my setup and I actually have folders like:
So i believe you should change the name of your directory to
and rerun metadata generation and you will be ready to go. |
|
Still no column file_path_mask_eroded_2_dilated_3. I'm going to carefully analyze the code and try again. Thank you very much. |
|
Ok, cool. Change the folder name by: mv ../masks_overlayed ../masks_overlayed_eroded_2_dilated_3But LEAVE the name in the masks_overlayed_dir: ../masks_overlayed/Rerun the metadata generation. I think @XYAskWhy got it to work pretty quickly. Any advice? |
|
Still not working... This is really a weird thing. |
|
Got the local training running using the older |
|
Master should work, dev too as i am generating final predictions with it right now. @XYAskWhy are you running unet_padding_tta ? Check evaluate checkpoint.py script to see how to add missing transformers (just run touch transformer_name in transformers dir) |
|
@jakubczakon Thanks for the tip. When I run
I looked into |
|
@apyskir, @taraspiotr can you provide some help here (check previous message). Thx :) |
|
@XYAskWhy when you run This issue will be solved by having |
|
@jakubczakon Thanks! Will check out soon. |
|
@jakubczakon Thanks for the update. But the README still doesn't include the |
|
@XYAskWhy it is included in the readme now. Thanks for spotting that. |
|
@jakubczakon is there a shorter way to evaluate model on test image/images or we have to prepare masks first which is time consuming. |
|
Well, you can simply use python main.py predict_on_dir \
--pipeline_name unet_tta_scoring_model \
--chunk_size 1000 \
--dir_path path/to/inference_directory \
--prediction_path path/to/predictions.jsonThat will get you predicted segmentation masks which you can later plot by using results exploration notebook. It is not a proper evaluation but it is definitely quicker. |
When running local pure python with
python main.py -- train_evaluate_predict --pipeline_name unet --chunk_size 5000, the following error occurs, any help?The text was updated successfully, but these errors were encountered: