Skip to content
A simple neural network for python autocompletion
Branch: master
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
parser ♻️ clean up code duplication Jul 16, 2019
.gitignore large dataset support Jul 10, 2019
.lab.yaml 🚀 release Jul 7, 2019
LICENSE Initial commit Jul 7, 2019
evaluate.py ♻️ fix Jul 16, 2019
evaluate_id.py ♻️ clean up code duplication Jul 16, 2019
extract_code.py
python-autocomplete.png 📸 screenshot Jul 7, 2019
readme.md 📚 markdown fix Jul 10, 2019
simple_model.py ♻️ fix Jul 16, 2019
train.py ♻️ fix Jul 16, 2019
train_id.py ♻️ fix Jul 16, 2019
vis_embeddings.py ♻️ move parsing classes Jul 16, 2019

readme.md

master branch is underdevelopment. Checkout simple_lstm branch to try it out.

This a toy project we started to see how well a simple LSTM model can autocomplete python code.

It gives quite decent results by saving above 30% key strokes in most files, and close to 50% in some. We calculated key strokes saved by making a single (best) prediction and selecting it with a single key.

We do a beam search to find predictions, upto ~10 characters ahead. So far it's too inefficient, if you are wondering about editor integration.

We train and predict on after cleaning comments, strings and blank lines in python code. The model is trained after tokenizing python code. It seems more efficient than character level prediction with byte-pair encoding.

A saved model is included in this repo. It is trained on tensorflow/models.

Here's a sample evaluation on a source file from validation set. Green characters are when a autocompletion started; i.e. user presses TAB to select the completion. The green character and and the following characters highlighted in gray are autocompleted. As you can see, it starts and ends completions arbitarily. That is a suggestion could be 'tensorfl' and not the complete identifier 'tensorflow' which can be a little annoying in a real usage scenario. We can limit them to finish on end of tokens to fix that. Also you can notice that it completes across operators as well. Increasing the length of the beam search will let it complete longer pieces of code.

Try it yourself

  1. Checkout simple_lstm branch.

  2. Setup lab

  3. Copy data to ./data/source

  4. Run extract_code.py to collect all python files, encode and merge them into all.py

  5. Run evaluate.py to evaluate the model.

  6. Run train.py to train the model

You can’t perform that action at this time.