How can I keep track of the methods' location?

[munybt]

This is what I would like to do:
Given a folder with source code files, generate a file with the vectors that represent each method, but instead of having only the name of the methods I would like some additional data to know the location of the methods (file name, start line, start column, end line, end column)

Is there any way I can achieve this output?

[urialon]

It's a little hacky, but you can achieve this by changing the JavaExtractor to add a unique ID to every method name, such that if the original name is "get|Foo" - the extractor will output "get|Foo|6153" and write the mapping from the ID "6153" to the metadata to another file or local database.
How does it sound?

[munybt]

Seems good, I'm gonna try it. Thank you :)

[munybt]

Hi @urialon ,
I implemented your suggestion, but now I am facing another problem.
For example, if I have a dataset with several different 'toString' methods, they will all have different target names, which will impact the training process because the names 'to|string|123' and 'to|string|456' will not match.

I tried to locate on the code a spot where I could change the equality comparison, but I couldn't find it (it think it is being handled by TensorFlow).

Can you help me with that?
Thank you very much

[urialon]

For training I don't think there is a point in doing that, I thought you wanted to do it at test time.
For training - train as usual

Once you have preprocessed test set (with unique IDs) and a trained model, you can change this line:

https://github.com/tech-srl/code2vec/blob/master/tensorflow_model.py#L176

such that the model will print original_names along with each original name's vector (currently it prints only the vectors in the same order as the examples in the test set).

[brash6]

Hello,
Is it possible to know in which order the preprocessing is done on code ?
For example, if I run preprocess.sh on several java files that each contain only one method and that method is always named 'f',
can I know in which order theses methods are preprocessed to appear in the c2v file ?
More precisely, can I know from which preprocessed file the i^th line of the c2v file comes from ?
Thank you !

[urialon]

Hi @brash6 ,
Currently, this is not implemented.
You can change the JavaExtractor code to print the file name that each line came from, and cut this field out before the next preprocessing step.

[brash6]

Thank you for you answer. I'll try this !