Branch: master
Find file History
Fetching latest commit…
Cannot retrieve the latest commit at this time.

Table of Contents

Alt Text

Each step in the above diagram corresponds to a Jupyter notebook in this repo. Below is a high level description of each step:

1 - Preprocess Data: describes how to get python files from BigQuery, and use the AST module to clean code and extract docstrings.

2 - Train Function Summarizer: build a sequence-to-sequence model to predict a docstring given a python function or method. The primary purpose of this model is for a transfer learning task that requires the extraction of features from code.

3 - Train Language Model: Build a language model using Fastai on a corpus of docstrings. We will use this model for transfer learning to encode short phrases or sentences, such as docstrings and search queries.

4 - Train Code2Emb Model: Fine-tune the model from step 2 to predict vectors instead of docstrings. This model will be used to represent code in the same vector space as the sentence embeddings produced in step 3. 

5 - Build Search Engine: Use the assets you created to created in steps 3 and 4 to create a semantic search tool.