Umbrella issue for ML team

[EgorBu]

Hello,
As we discussed in slack here it's umbrella issue for ML team. I will try to aggregate our wishes and problems here.

Issues

Let's start with issues:

bblfshd issues

• A lot of <defunct> #159 A lot of <defunct>
• Cannot run on Windows #164 Cannot run on Windows
• A lot of files in /var/lib/bblfshd/tmp/ #168 A lot of files in /var/lib/bblfshd/tmp/
• Downgrading drivers doesn't yield functional drivers. #184 Downgrading drivers doesn't yield functional drivers.
• Memory is not released, number of instances in bblfshctl driver list mismatch with htop #219 Memory is not released, number of instances in bblfshctl driver list mismatch with htop
• Cannot install Go driver #230 Cannot install Go driver
• (FR) Add Babelfishfile #222 Add Babelfishfile
• (FR) Add a way to reinstall the driver with one command #197 Add a way to reinstall the driver with one command

Go & Python client issues

• https://github.com/bblfsh/client-go/issues/100 Question: how to get the node's token and internal type
• Queries are slow (v1) python-client#101 [performance] queries are slow
• ERROR:root:Exception deserializing message! python-client#89 ERROR:root:Exception deserializing message!
• Function supported_languages raises unimplemented exception python-client#125 Function supported_languages raises unimplemented exception
• language argument in client.parse changes number of files signigicantly python-client#126 language argument in client.parse changes number of files significantly
• (FR) [feature request] add position information python-client#108 Add position information

Drivers

• (FR) Comments are not merged go-driver#13 Comments are not merged
• [bug] Missed token for regular expressions javascript-driver#22 [bug] Missed token for regular expressions
• [bug] roles: UNANNOTATED javascript-driver#24 [bug] roles: UNANNOTATED
• Multibyte characters break positions javascript-driver#25 Multibyte characters break positions
• \n should not be parsed in strings javascript-driver#32 \n should not be parsed in strings
• On EndPosition-s again python-driver#142 On EndPosition-s again
• Errors while extracting all commits in pallets/jinja python-driver#143 Errors while extracting all commits in pallets/jinja
• Errors during installation of drivers python-driver#156 Errors during installation of drivers
• (FR) Support Cython python-driver#150 Support Cython
• Error, malformed driver, missing manifest.toml ocaml-driver#8 Error, malformed driver, missing manifest.toml
• Error, malformed driver, missing manifest.toml rust-driver#8 Error, malformed driver, missing manifest.toml

SDK

• (FR) Add parent field into UAST Node in clients. sdk#252 Add parent field into UAST Node in clients.
• (FR) [feature] Include the information about full tokens sdk#291 Include the information about full tokens

Web

• (FR) Add installed drivers versions info web#158 Add installed drivers versions info
• (FR) Add tree-like visualization web#42 Add tree-like visualization

Miscellaneous

• Babelfish (aka. bblfsh) link is broken documentation#199 Babelfish (aka. bblfsh) link is broken
• (FR) Feature request: node hashes libuast#78 Feature request: node hashes

I believe that some of these issues can be closed already.

Wishlist

Let's start second part about wishes (probably it's a bit late for new year's resolution but anyway).
It should show our priorities.

Feature extraction for source code

This part is very important for everybody in MLonCode area and still it's quite complicated to do.
In summary:

• tokenization: code -> nodes -> "".join(nodes) == code
• parent information Add parent field into UAST Node in clients. sdk#252 - we use this info, these people use this info too https://github.com/tech-srl/ - this feature may increase usage of bblfsh by researchers in this area
• reserved keywords
• roles that are used in particular language

What we need (and not only we) - code -> UAST -> code. It's not possible right now so we did several tricks to overcome this problem.

We wrote tokenizer for source code (right now only JS-lang is supported) that allows to do code -> {UAST, info about parent of nodes, tokens} where tokens has property "".join(tokens) == code.

To do it we had to find reserved keywords for particular language and made logic to split everything between UAST nodes with tokens into reserved keywords and spaces/quotes/newlines.

Also it's important to know which roles are used in particular language (obviously only subset of all roles is used for some language -> and it can be used to reduce dimensionality of features like we did here)

Position information

We rely heavily on position information of identifiers and literals - if it's not available - it makes our life much harder.

So we have to make workarounds like this for [bug] Missed token for regular expressions.

How it can be checked - make long tests for each driver for 10-20-30 best repositories for each language and check that node.token == code[node.start_offset:node.end_offset]

Integration/regression tests on popular repositories

Maybe proces the top 10-20-30 repositories for each supported language.

Make long running tests to check if everything is good with drivers-sdk-etc - it will allow to avoid many problems like

• A lot of <defunct> #159 A lot of <defunct>
• Memory is not released, number of instances in bblfshctl driver list mismatch with htop #219 Memory is not released, number of instances in bblfshctl driver list mismatch with htop (probably related to A lot of <defunct> #159).
• [bug] Missed token for regular expressions javascript-driver#22 [bug] Missed token for regular expressions
• [bug] roles: UNANNOTATED javascript-driver#24 [bug] roles: UNANNOTATED
• Multibyte characters break positions javascript-driver#25 Multibyte characters break positions

and so on.

Language extensions

Follow up of this discussion https://src-d.slack.com/archives/C7UDG9VNY/p1533802242000193. In summary: https://github.com/babel/babel/ is excluded from our list of repos to check because more an issue of the native driver not covering all node types in this repo. Probably it's driver/enry issue to detect such kind of language extensions. (another example https://github.com/samgozman/YoptaScript#%D0%9F%D1%80%D0%B8%D0%BC%D0%B5%D1%80%D1%8B)

Partial parsing

It could help to check that code wasn't changed in the way to break parsing of UAST. Right now it just parse full content of file even if one space is added.

[creachadair]

Thank you very much for writing this up, @EgorBu. There's a lot to consider here, but this will be really helpful for our planning.

[zurk]

+ https://github.com/bblfsh/bblfshd/issues/236 (JS driver related)

[dennwc]

2) was fixed in bblfsh/javascript-driver#50

[dennwc]

I've checked the issues list and some of them are already resolved.

@creachadair Maybe we can make a separate project for this to track the progress?

[creachadair]

This would make a good milestone. I'll set that up.

Edit: I started doing this, but it wound up opening up various design questions I didn't want to pre-answer, so I went with the simpler approach @bzz suggested below for now.

[bzz]

Short-term and as a first step we could convert PR description to have a checkboxes and mark issues that are solved. Happy to do that, if nobody has any objections or better short-term suggestions.

[creachadair]

Short-term and as a first step we could convert PR description to have a checkboxes and mark issues that are solved. Happy to do that, if nobody has any objections or better short-term suggestions.

It's a good idea, and I started working on this. So far I have mostly just rearranged things cosmetically, checking off items that are complete and separating out the FRs from the bugs.

[bzz]

Did a pass over checkboxes in Description and updated resolved ones.

[dennwc]

I think we can close this one, assuming all feature requests have their own issue filed.

[creachadair]

I think we can close this one, assuming all feature requests have their own issue filed.

It looks that way. If I'm wrong, let me know and I'll take care of it.