MAX_CONTEXTS tied to preprocessing?

[hsellik]

Hi @urialon ,

I am trying to do some hyper-parameter tuning, but it seems it is a bit trickier than I thought.

1. In order to change MAX_CONTEXTS in config.py, do I have to preprocess the data with the same MAX_CONTEXTS value as well?

2. Does this also apply for WORD_VOCAB_SIZE and PATH_VOCAB_SIZE since I see correspondence in preprocessing and running:
   WORD_VOCAB_SIZE == config.MAX_TOKEN_VOCAB_SIZE
   PATH_VOCAB_SIZE == config.MAX_PATH_VOCAB_SIZE

[urialon]

Hi @hsellik ,

1. As far as I remember - no, you don't need to preprocess the data yourself, as long as the number of contexts that you wish to use is lower or equal than the number of contexts that the data was preprocessed with. The number of contexts in the data (i.e., the way the data was preprocessed) is saved in the data dictionary, and then loaded here. If I remember correctly, the data was preprocessed with 1000 contexts. Thus, you don't need to re-preprocess the data, you can just change MAX_CONTEXTS here and the reader will automatically sample MAX_CONTEXTS out of the total 1000 that were saved with the data.

2. You don't need to re-preprocess the data here as well. The data was saved with some really large vocabulary sizes (to check the actual numbers, put a breakpoint here and check the len of the loaded dictionaries). If you reduce vocab sizes here, the code will automatically take only the most frequent values.

E.g., if the data was preprocessed with a vocabulary of 1M and you set the vocab size to 1K - then the code will load all 1M values, sort them (descendingly) by their frequency, and take only the top 1K.

If you wish to use larger values for contexts or vocabularies than the values that the data was preprocessed with -- then yes, in that case, you will have to re-preprocess the data. However, I don't think that using more than 1000 contexts or larger vocabularies will help.

I hope it helps, let me know if I was unclear or if you have any other questions.

[hsellik]

Okay, makes perfect sense, does the same apply for code2vec project?

[urialon]

In code2vec -
Regarding vocabularies - yes - you can reduce the sizes.
Regarding num_contexts - I don't think that you can change that

[hsellik]

Okay, nice and clear. Thank you for the quick answers! :)

[Guardian99]

On running a debugger, I get a message of "Expect 1001 fields but have 2456 in record 0".
1)How do i deal with this error in general, If i have to process codes that have approximately 2500-3000 lines.