Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

data problem on Polymorphic Transformers #39

Open
kathyliu579 opened this issue Jun 7, 2022 · 14 comments
Open

data problem on Polymorphic Transformers #39

kathyliu579 opened this issue Jun 7, 2022 · 14 comments

Comments

@kathyliu579
Copy link

Hi~please help me figure out some questions.

  1. i found that " python3 train2d.py --task refuge --ds train,valid,test --split all --maxiter 10000 --net unet-scratch "
    should be " --task fundus", otherwise will report errors.
  2. the data for polyp downloaded from https://github.com/DengPingFan/PraNet (search for "testing data") is not complete. some image is missing, i guess it should include in their training data. But for training data, they have several datasets mixed together into two folders(image and mask)..so should i manually select image to our folders? Could you have a look?

many thanks in advance
@kathyliu579
Copy link
Author

and also i found when i run the 2 th step of fundus.
there has some problems:

(torch17) qianying@merig:~/PycharmProjects/segtran-master/code$ python3 train2d.py --split all --maxiter 3000 --task fundus --net unet-scratch --ds train,valid,test --polyformer source --cp ../model/unet-scratch-refuge-train,valid,test-06072104/iter_7000.pth --sourceopt allpoly

Traceback (most recent call last):
  File "train2d.py", line 939, in <module>
    net = VanillaUNet(n_channels=3, num_classes=args.num_classes, 
  File "/home/qianying/PycharmProjects/segtran-master/code/networks/unet2d/unet_model.py", line 32, in __init__
    self.polyformer = Polyformer(feat_dim=64, args=polyformer_args)
  File "/home/qianying/PycharmProjects/segtran-master/code/networks/polyformer.py", line 117, in __init__
    polyformer_layers.append( PolyformerLayer(str(i), config) )
  File "/home/qianying/PycharmProjects/segtran-master/code/networks/polyformer.py", line 24, in __init__
    self.in_ator_trans  = CrossAttFeatTrans(config, name + '-in-squeeze')
  File "/home/qianying/PycharmProjects/segtran-master/code/networks/segtran_shared.py", line 502, in __init__
    self.out_trans  = ExpandedFeatTrans(config,  name)
  File "/home/qianying/PycharmProjects/segtran-master/code/networks/segtran_shared.py", line 344, in __init__
    if not config.use_mince_transformer or config.mince_scales is None:
AttributeError: 'EasyDict' object has no attribute 'use_mince_transformer'

and i found these term show in before.

############## Mince transformer settings ##############
parser.add_argument("--mince", dest='use_mince_transformer', action='store_true',
                    help='Use Mince (Multi-scale) Transformer to save GPU RAM.')
parser.add_argument("--mincescales", dest='mince_scales', type=str, default=None, 
                    help='A list of numbers indicating the mince scales.')
parser.add_argument("--minceprops", dest='mince_channel_props', type=str, default=None, 
                    help='A list of numbers indicating the relative proportions of channels of each scale.')

emmm so what happens?

@kathyliu579
Copy link
Author

ohh i saw the previous issue.

 # if not config.use_mince_transformer or config.mince_scales is None:
        self.num_scales     = 0
        self.mince_scales   = None
        self.mince_channels = None
        # else:
        #     # mince_scales: [1, 2, 3, 4...]
        #     self.mince_scales   = config.mince_scales
        #     self.num_scales     = len(self.mince_scales)
        #     self.mince_channel_props = config.mince_channel_props
        #     self.mince_channel_indices, mince_channel_nums = \
        #         fracs_to_indices(self.feat_dim, self.mince_channel_props)

now i revise it like this. is it right??

and happens some other errors..

File "/home/qianying/PycharmProjects/segtran-master/code/networks/segtran_shared.py", line 506, in __init__
    self.keep_attn_scores = config.use_attn_consist_loss
AttributeError: 'EasyDict' object has no attribute 'use_attn_consist_loss'

how to fix it?

@kathyliu579
Copy link
Author

also have a question on fine tune on "k".
A polyformer layer consists of two sub-transformers 1 and 2. Does this paper only finetune the k of sub-transformers 1?
cause in the code, i only see:

            for poly_opt_mode in poly_opt_modes:
                if poly_opt_mode == 'allpoly':
                    optimized_params += [ translayers.named_parameters() ]
                elif poly_opt_mode == 'inator':
                    optimized_params += [ translayer.in_ator_trans.named_parameters() for translayer in translayers ]
                elif poly_opt_mode == 'k':
                    optimized_params += [ translayer.in_ator_trans.key.named_parameters()   for translayer in translayers ]
                elif poly_opt_mode == 'v':
                    optimized_params += [ translayer.in_ator_trans.out_trans.first_linear.named_parameters() for translayer in translayers ]
                elif poly_opt_mode == 'q':
                    optimized_params += [ translayer.in_ator_trans.query.named_parameters() for translayer in translayers ]

and the in_ator_trans is the sub-transformers 1, right?

@askerlee
Copy link
Owner

askerlee commented Jun 9, 2022

Thanks for reporting the bug. I've just corrected "refuge" to "fundus". Also I've simplified the polyformer config.
Yes you are right. Fine-tuning k is only to finetune the k of sub-transformer 1.

@kathyliu579
Copy link
Author

kathyliu579 commented Jun 9, 2022 via email

@kathyliu579
Copy link
Author

kathyliu579 commented Jun 9, 2022 via email

@askerlee
Copy link
Owner

askerlee commented Jun 9, 2022 •
edited
Loading

“Also I've simplified the polyformer config.” so what file i should replace？

You can just do a "git pull origin master" to update the code.

in addition may i ask why in training, q k are shared ？

Yes correct. It's explained in the IJCAI paper, page 3:

  • In traditional transformers, the key and query projections are independently learned, enabling them to capture asymmetric relationships between tokens in natural language. However, the relationships between image units are often symmetric, such as whether two pixels belong to the same segmentation class... In both blocks, the query projections and key projections are tied to make the attention symmetric, for better modeling of the symmetric relationships between image units.
  • Empirically, tying q and k leads to better performance on medical image segmentation (I haven't tried more general segmentation tasks other than medical).

why only finetune k of transformer 1?

Because empirically when I just fine-tune k of transformer 1, it already performs well. I didn't try to fine-tune both, and I'm not sure how to intuitively understand the benefits of fine-tuning both layers for domain adaptation.

@askerlee
Copy link
Owner

askerlee commented Jun 9, 2022

in addition, the poly dataset i downloaded has different folders... can you please upload your processed data？

You mean polyp?
For people in China:
https://pan.baidu.com/s/1TuiPyQirN4J2hQfQxMMHkQ?pwd=whjl
For people in other countries:
https://www.dropbox.com/s/s5v2kotxtvruigp/polyp.tar?dl=0

@kathyliu579
Copy link
Author

kathyliu579 commented Jun 9, 2022 via email •
edited
Loading

@kathyliu579
Copy link
Author

in addition, can you update the commands of polyp dataset ?
i am not sure for the commands of step 3 and 4 (training and test on target domain). can you have a look? if it is right, you can add to the "read me".

python3 train2d.py --task polyp --ds CVC-300 --split train --samplenum 5 --maxiter 1600 --saveiter 40 --net unet-scratch --cp ../model/unet-scratch-polyp-CVC-ClinicDB-train,Kvasir-train-06101057/iter_500.pth --polyformer target --targetopt k --bnopt affine --adv feat --sourceds CVC-ClinicDB-train,Kvasir-train --domweight 0.002 --bs 3 --sourcebs 2 --targetbs 2

especially for the "sourceds ", i am not sure.

python3 test2d.py --gpu 1 --ds CVC-300--split test --samplenum 5 --bs 6 --task polyp –cpdir .. --net unet-scratch --polyformer target --nosave --iters 40-1600,40

especially for the "split".

@kathyliu579
Copy link
Author

Also I have other 2 questiones.

  1. I noticed that only ClinicDB and Kvasir's training set seems to be used for training. the test dataset we donot use?
  2. why for fundus we trained on source data "ds"= train,valid,test, but when train on target domain (step 3), "sourceds" only includes "train", not "train,valid,test"?

@nguyenlecong
Copy link

Also I have other 2 questiones.

  1. I noticed that only ClinicDB and Kvasir's training set seems to be used for training. the test dataset we donot use?
  2. why for fundus we trained on source data "ds"= train,valid,test, but when train on target domain (step 3), "sourceds" only includes "train", not "train,valid,test"?

Excuse me, do you have a problem with loss function when training polyformer (source) as shown?
image

If not, can you show the loss function and the command line you use?
This is my command line:
!python3 train2d.py --task polyp --split all --maxiter 3000 --net unet-scratch --polyformer source --modes 2 --ds CVC-ClinicDB-train,Kvasir-train --cp ../model/unet-scratch-polyp-CVC-ClinicDB-train,Kvasir-train-06111827/iter_14000.pth --sourceopt allpoly
Thank you!

@kathyliu579
Copy link
Author

Also I have other 2 questiones.

  1. I noticed that only ClinicDB and Kvasir's training set seems to be used for training. the test dataset we donot use?
  2. why for fundus we trained on source data "ds"= train,valid,test, but when train on target domain (step 3), "sourceds" only includes "train", not "train,valid,test"?

Excuse me, do you have a problem with loss function when training polyformer (source) as shown? image

If not, can you show the loss function and the command line you use? This is my command line: !python3 train2d.py --task polyp --split all --maxiter 3000 --net unet-scratch --polyformer source --modes 2 --ds CVC-ClinicDB-train,Kvasir-train --cp ../model/unet-scratch-polyp-CVC-ClinicDB-train,Kvasir-train-06111827/iter_14000.pth --sourceopt allpoly Thank you!

Hi may i ask how you draw the loss fuction? i have not seen it. if you tell, i can check mine.

@nguyenlecong
Copy link

Also I have other 2 questiones.

  1. I noticed that only ClinicDB and Kvasir's training set seems to be used for training. the test dataset we donot use?
  2. why for fundus we trained on source data "ds"= train,valid,test, but when train on target domain (step 3), "sourceds" only includes "train", not "train,valid,test"?

Excuse me, do you have a problem with loss function when training polyformer (source) as shown? image
If not, can you show the loss function and the command line you use? This is my command line: !python3 train2d.py --task polyp --split all --maxiter 3000 --net unet-scratch --polyformer source --modes 2 --ds CVC-ClinicDB-train,Kvasir-train --cp ../model/unet-scratch-polyp-CVC-ClinicDB-train,Kvasir-train-06111827/iter_14000.pth --sourceopt allpoly Thank you!

Hi may i ask how you draw the loss fuction? i have not seen it. if you tell, i can check mine.

You can load log from directory ../model/*/log/event... by:
from tensorboard.backend.event_processing import event_accumulator
ea = event_accumulator.EventAccumulator(logdir)
ea.Reload()
loss = pd.DataFrame(ea.Scalars('loss/loss'))
There are also loss/total_ce_loss and loss/total_dice_loss

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@askerlee @nguyenlecong @kathyliu579