How to deal with long code data?

[jackfromeast]

I am using codeBERT for classifying malicious code written in PHP. Some codes in the dataset are really long which far beyond the general MAX_LEN of a sentence for example 256. And setting MAX_LEN a big number would soon result in GPU resources exhausted. So I wonder if there are some fine strategies to deal with it.

[guoday]

Our model only supports 512 max length. One strategy is that you can split a code into K segments, and then feed K segments into CodeBERT separately. Finally, you average or use a RNN over K representations of CodeBERT to obtain a global representation.

[jackfromeast]

Thankyou! It did solve my problem!

[Silverhorse7]

I am using CodeBERT embeddings in my module, does the result embeddings get affected by the token size too? @guoday