Some help with reproduction

[bruAristimunha]

Hello @935963004,

I would like to starting say thank you for your work, I think it is a fundamental and necessary work in EEG decoding. Thank you for that!

So, I am trying to understand and run your code, but some things are not working, and I would like to request your assistance. From the beginning, with a toy example.

import torch
from torch import nn

from modeling_finetune import NeuralTransformer

# As commment in the meet, the expect input is:
# Batch size, channels, time//patch_size, patch_size

in_chans = 1  # **Not working if in_chans is different of 1. Issue with temporal_embedding.**
batch_size = 1
patch_size = 200
n_time_points_patched = 16  # Max number for patch, the value is hardcode in
# the model
EEG_size = 1600

# Generating an empty vector just to get the output.
X = torch.zeros(batch_size, in_chans, n_time_points_patched, patch_size)
# Everything is default
model = NeuralTransformer(
    EEG_size=EEG_size,
    patch_size=patch_size,
    in_chans=in_chans,
    out_chans=8,
    num_classes=1000,
    embed_dim=200,
    depth=12,
    num_heads=10,
    mlp_ratio=4.,
    qkv_bias=False,
    qk_norm=None,
    qk_scale=None,
    drop_rate=0.,
    attn_drop_rate=0.,
    drop_path_rate=0.,
    norm_layer=nn.LayerNorm,
    init_values=0, # default value is not working, changed from None to zero.
    use_abs_pos_emb=False,  # Not working
    use_rel_pos_bias=False, 
    use_shared_rel_pos_bias=False,
    use_mean_pooling=True,
    init_scale=0.001,
)

with torch.no_grad():
    y_pred = model(X)

My questions are:

• How to make it work with any number of channels?
• How do we solve the issue with positional embedding? And what about temporal embedding?
• How to adapt the model to get something as input:
  "(batch, channel, time_steps)"

In my naive intuition if I change the in_chans everything should working because of the TemporalConv module, but it's not.

FYI @LemonFace0309, @jonxuxu and @shahbuland, @RashikShahjahan

All the best!

[bruAristimunha]

Small update @935963004, I think we solved the issue with a number of chans, and positional embedding.. (I opened a PR).

We are still facing problems with use_rel_pos_bias and use_shared_rel_pos_bias

[935963004]

I think you should set use_abs_pos_emb=True, use_rel_pos_bias=False, use_shared_rel_pos_bias=False. Does this work for you?

[bruAristimunha]

Yes @935963004, with PR #2 it will work, but I don't really understand the reason for having options that aren't used anywhere. And I am not sure if the modification are okay for you.

Another thing I was thinking about is how it is building the patch... Now there is no patch construction within the network, i.e. the network already expects the input [batch, n_chans, num_patch, patch size], why this it is not learned during the train, as in the ViT or BIET (1, 2 or 3)?

I really appreciate your input on this! 🙏

[935963004]

The input x is [batch, n_chans, num_patch, patch size]. In the TemporalConv, x is first transformed to [batch, n_chans * num_patch, patch size]. Then, for using the torch.nn.Conv2d(), x is unsqueezed to [batch, 1, n_chans * num_patch, patch size], where 1 is the in_chans, just like rgb for images, so it is fixed and can not be changed. After several convolutional layers, x will be transformed back to [batch, n_chans * num_patch, patch size], which can be passed into the Transformer encoder as input. Can this explanation help you?

[RashikShahjahan]

@935963004 Is the input channels always expected to be 1 ? Because, in the code the TemporalConv is only called for in_chan =1.

[935963004]

@935963004 Is the input channels always expected to be 1 ? Because, in the code the TemporalConv is only called for in_chan =1.

Exactly

[RashikShahjahan]

@935963004 I am a bit confused, what about multi-channel EEGs?

[935963004]

@935963004 I am a bit confused, what about multi-channel EEGs?

The input x [batch, n_chans, num_patch, patch size] is multi-channel EEG. in_chan and n_chans are two things. in_chan is just for convolution operation thus it is set to 1 (actually we just reshape the original input from [batch, n_chans, num_patch, patch size] to [batch, 1, n_chans * num_patch, patch size]), while n_chans is the number of electrodes for multi-channel EEG.

[RashikShahjahan]

@935963004 Thank you very much! @bruAristimunha I guess we are good without my changes then.

[bruAristimunha]

Ok, thanks @935963004 and @RashikShahjahan!

Last thing for me, I was wondering, can you please clean up the code a little or put a doc string inside the model?

The names of the variables within the model are not super obvious, and I'm pretty sure it will lead other users to open more issues or send emails to you or to the rest of the authors.

I truly understand and empathize with all the effort you've made with your model, and also understand that during development some decisions are not always optimized. however, I would like to thank you in advance for any effort you can make to ensure a more easy reproduction.

Have a nice day!

[935963004]

I'm sorry for the inconvenience and I appreciate your suggestion. I will add some annotations for better understanding in the following days.

[bruAristimunha]

Hey @935963004,

I have some more questions for you:

In the temporal embedding, you define the temporal embedding with a space of 16 items, what is the reason for choosing this number? I couldn't find it anywhere in the code, or paper. It looks like you've always had the same number of patches, is this correct? It seems like it's linked to the number of patches, but I'm not sure.

The same question for position embedding. It seems like there are always 128 positions, I couldn't understand the math to arrive at these numbers.

https://github.com/935963004/LaBraM/blob/main/modeling_finetune.py#L283

[935963004]

Hey @935963004,

I have some more questions for you:

In the temporal embedding, you define the temporal embedding with a space of 16 items, what is the reason for choosing this number? I couldn't find it anywhere in the code, or paper. It looks like you've always had the same number of patches, is this correct? It seems like it's linked to the number of patches, but I'm not sure.

The same question for position embedding. It seems like there are always 128 positions, I couldn't understand the math to arrive at these numbers.

https://github.com/935963004/LaBraM/blob/main/modeling_finetune.py#L283

These numbers are set to meet the maximum requirements of our paper. In fact, you can set them to any number if you like as long as they meet your maximum requirements.

[MisterKloudy]

Hi all, I'm also facing problems with reproduction.
I am currently not working on the TUH EEG datasets but am hoping to be able to use the LaBraM embeddings for other BCI tasks.
What is the proper format for inputs to the dataset maker and to the model?

[935963004]

Hi all, I'm also facing problems with reproduction. I am currently not working on the TUH EEG datasets but am hoping to be able to use the LaBraM embeddings for other BCI tasks. What is the proper format for inputs to the dataset maker and to the model?

There are various ways for you to implement with your own dataset. Just make sure the dataloader and ch_names fit our implementation. You can refer to run_class_finetuning.py and replace the get_dataset function with your own one.

[MisterKloudy]

I'm getting an error on the positional embedding when using the default settings.
Do we need to specify any additional parameters when calling the run_class_finetuning.py script?

pos_embed_used = self.pos_embed[:, input_chans] if input_chans is not None else self.pos_embed
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
TypeError: 'NoneType' object is not subscriptable

[935963004]

I'm getting an error on the positional embedding when using the default settings. Do we need to specify any additional parameters when calling the run_class_finetuning.py script?

pos_embed_used = self.pos_embed[:, input_chans] if input_chans is not None else self.pos_embed ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^ TypeError: 'NoneType' object is not subscriptable

I think you should set abs_pos_emb to True in args. You are recommended to use the provided script in README:
OMP_NUM_THREADS=1 torchrun --nnodes=1 --nproc_per_node=8 run_class_finetuning.py
--output_dir ./checkpoints/finetune_tuab_base/
--log_dir ./log/finetune_tuab_base
--model labram_base_patch200_200
--finetune ./checkpoints/labram-base.pth
--weight_decay 0.05
--batch_size 64
--lr 5e-4
--update_freq 1
--warmup_epochs 5
--epochs 50
--layer_decay 0.65
--drop_path 0.1
--dist_eval
--save_ckpt_freq 5
--disable_rel_pos_bias
--abs_pos_emb
--dataset TUAB
--disable_qkv_bias
--seed 0

[upper127]

I'm getting an error on the positional embedding when using the default settings. Do we need to specify any additional parameters when calling the run_class_finetuning.py script?

pos_embed_used = self.pos_embed[:, input_chans] if input_chans is not None else self.pos_embed ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^ TypeError: 'NoneType' object is not subscriptable

Have you made any progress, in terms of processing your own dataset?

[MisterKloudy]

I'm getting an error on the positional embedding when using the default settings. Do we need to specify any additional parameters when calling the run_class_finetuning.py script?
pos_embed_used = self.pos_embed[:, input_chans] if input_chans is not None else self.pos_embed ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^ TypeError: 'NoneType' object is not subscriptable

Have you made any progress, in terms of processing your own dataset?

Yes! With some tweaks to the provided script I was able to process my own dataset. I was working on the MindBigData dataset but unfortunately I was not able to get good results for the task in the dataset. I think it could be due to insufficient signals in the dataset for the task, or that the embeddings were not suitable for the dataset. I was only able to get about 30+% accuracy in a 10-class classification. Better than pure chance but not good enough for anything major I think.

[upper127]

I'm getting an error on the positional embedding when using the default settings. Do we need to specify any additional parameters when calling the run_class_finetuning.py script?
pos_embed_used = self.pos_embed[:, input_chans] if input_chans is not None else self.pos_embed ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^ TypeError: 'NoneType' object is not subscriptable

Have you made any progress, in terms of processing your own dataset?

Yes! With some tweaks to the provided script I was able to process my own dataset. I was working on the MindBigData dataset but unfortunately I was not able to get good results for the task in the dataset. I think it could be due to insufficient signals in the dataset for the task, or that the embeddings were not suitable for the dataset. I was only able to get about 30+% accuracy in a 10-class classification. Better than pure chance but not good enough for anything major I think.

I think this may be because the raw model doesn't involve your own tasks, so the accuracy leaves something to be desired, try pre-training with your own tasks. What should I do to input my own data into the original model? Take the cnt file to do the categorization think for example.

[Enzo2357]

使用默认设置时，我在位置嵌入时遇到错误。调用 run_class_finetuning.py 脚本时是否需要指定任何其他参数？ pos_embed_used = self.pos_embed[：， input_chans] 如果 input_chans 不是 None else self.pos_embed ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^ TypeError： 'NoneType' object is not subscriptable

在处理自己的数据集方面，您有没有取得任何进展？

是的！通过对提供的脚本进行一些调整，我能够处理自己的数据集。我正在处理MindBigData数据集，但不幸的是，我无法在数据集中获得良好的任务结果。我认为这可能是由于数据集中没有足够的信号来完成任务，或者嵌入不适合数据集。在 30 类分类中，我只能获得大约 10+% 的准确率。比纯粹的机会要好，但我认为对于任何重大的事情来说都不够好。

您好，可以告诉我一下您是如何调整的吗？

[MisterKloudy]

使用默认设置时，我在位置嵌入时遇到错误。调用 run_class_finetuning.py 脚本时是否需要指定任何其他参数？ pos_embed_used = self.pos_embed[：， input_chans] 如果 input_chans 不是 None else self.pos_embed ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^ TypeError： 'NoneType' object is not subscriptable

在处理自己的数据集方面，您有没有取得任何进展？

是的！通过对提供的脚本进行一些调整，我能够处理自己的数据集。我正在处理MindBigData数据集，但不幸的是，我无法在数据集中获得良好的任务结果。我认为这可能是由于数据集中没有足够的信号来完成任务，或者嵌入不适合数据集。在 30 类分类中，我只能获得大约 10+% 的准确率。比纯粹的机会要好，但我认为对于任何重大的事情来说都不够好。

您好，可以告诉我一下您是如何调整的吗？

我是先把自己的数据集转换成Signal和label然后自己定义了一些新的Dataloader

class MIND2BLoader(torch.utils.data.Dataset):
    def __init__(self, root, files, sampling_rate=128):
        self.root = root
        self.files = files
        self.default_rate = 128
        self.sampling_rate = sampling_rate

    def __len__(self):
        return len(self.files)

    def __getitem__(self, index):
        sample = pickle.load(open(os.path.join(self.root, self.files[index]), "rb"))
        X = sample["signal"]
        Y = int(sample["label"])
        X = torch.FloatTensor(X)
        return X, Y

def prepare_MIND_2B_dataset(root):
    # set random seed
    seed = 4523
    np.random.seed(seed)

    train_files = os.listdir(os.path.join(root, "train"))
    val_files = os.listdir(os.path.join(root, "val"))
    test_files = os.listdir(os.path.join(root, "test"))

    # prepare training and test data loader
    train_dataset = MIND2BLoader(
        os.path.join(
            root, "train"), train_files, sampling_rate=128
    )
    test_dataset = MIND2BLoader(
        os.path.join(
            root, "val"), test_files, sampling_rate=128
    )
    val_dataset = MIND2BLoader(
        os.path.join(
            root, "test"), val_files, sampling_rate=128
    )
    print(len(train_files), len(val_files), len(test_files))
    return train_dataset, test_dataset, val_dataset
    
    ```

[upper127]

使用默认设置时，我在位置嵌入时遇到错误。调用 run_class_finetuning.py 脚本时是否需要指定任何其他参数？ pos_embed_used = self.pos_embed[：， input_chans] 如果 input_chans 不是 None else self.pos_embed ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^ TypeError： 'NoneType' object is not subscriptable

在处理自己的数据集方面，您有没有取得任何进展？

是的！通过对提供的脚本进行一些调整，我能够处理自己的数据集。我正在处理MindBigData数据集，但不幸的是，我无法在数据集中获得良好的任务结果。我认为这可能是由于数据集中没有足够的信号来完成任务，或者嵌入不适合数据集。在 30 类分类中，我只能获得大约 10+% 的准确率。比纯粹的机会要好，但我认为对于任何重大的事情来说都不够好。

您好，可以告诉我一下您是如何调整的吗？

我是先把自己的数据集转换成Signal和label然后自己定义了一些新的Dataloader

class MIND2BLoader(torch.utils.data.Dataset):
    def __init__(self, root, files, sampling_rate=128):
        self.root = root
        self.files = files
        self.default_rate = 128
        self.sampling_rate = sampling_rate

    def __len__(self):
        return len(self.files)

    def __getitem__(self, index):
        sample = pickle.load(open(os.path.join(self.root, self.files[index]), "rb"))
        X = sample["signal"]
        Y = int(sample["label"])
        X = torch.FloatTensor(X)
        return X, Y

def prepare_MIND_2B_dataset(root):
    # set random seed
    seed = 4523
    np.random.seed(seed)

    train_files = os.listdir(os.path.join(root, "train"))
    val_files = os.listdir(os.path.join(root, "val"))
    test_files = os.listdir(os.path.join(root, "test"))

    # prepare training and test data loader
    train_dataset = MIND2BLoader(
        os.path.join(
            root, "train"), train_files, sampling_rate=128
    )
    test_dataset = MIND2BLoader(
        os.path.join(
            root, "val"), test_files, sampling_rate=128
    )
    val_dataset = MIND2BLoader(
        os.path.join(
            root, "test"), val_files, sampling_rate=128
    )
    print(len(train_files), len(val_files), len(test_files))
    return train_dataset, test_dataset, val_dataset
    
    ```

test_dataset = MIND2BLoader(
os.path.join(
root, "val"), test_files, sampling_rate=128
)
val_dataset = MIND2BLoader(
os.path.join(
root, "test"), val_files, sampling_rate=128

        这两个文件夹是命名存在错误码， val 和test 

[upper127]

可以了解你一下你使用的数据标签类型是怎么设置的吗，是从0开始的吗（我的标签是四种 1 2 3 4）

[MisterKloudy]

是的 0 到 9

…

On Wed, May 1, 2024, 10:59 AM upper127 ***@***.***> wrote: 可以了解你一下你使用的数据标签类型是怎么设置的吗，是从0开始的吗（我的标签是四种 1 2 3 4） — Reply to this email directly, view it on GitHub <#1 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BGTHY26HY7Q4PG6AN4N4HL3ZABLALAVCNFSM6AAAAABDPQ77YCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBXHEYDGNBXGA> . You are receiving this because you commented.Message ID: ***@***.***>