Fast trigram based code search
Switch branches/tags
Nothing to show
Clone or download
keegancsmith and hanwen zoekt-archive-index: Add isGitOID
This improves the heuristic for deciding if a reference is a commit or a branch name.

Change-Id: I527e87c1a593ce552f9b4c0e74cfc344540b8364
Latest commit 8e284ca May 29, 2018
Permalink
Failed to load latest commit information.
build Change IsText to CheckText returning a specific skip reason May 25, 2018
cmd zoekt-archive-index: Add isGitOID May 30, 2018
ctags ctags: Skip TestJSON if universal-ctags is missing Apr 18, 2018
doc doc: add ctags compilation requirements Apr 16, 2018
gitindex Set SkipReason when encountering large blobs in git May 25, 2018
query query: Apply case to all children Feb 8, 2018
shards web: add 'order' query param to list repos Apr 19, 2018
web web: fix broken </table> in repolist template May 23, 2018
.gitignore Update gitignore for new binary names. Jul 28, 2016
CONTRIBUTING add a note about tests in CONTRIBUTING Jan 22, 2018
LICENSE Add boilerplate. Apr 7, 2016
README.md doc: add more doc on compiling and installing universal-ctags Jan 29, 2018
all.bash Use /... package selector in all.bash Apr 18, 2018
api.go Add IndexMetadata.Version, populated by the link step May 17, 2018
bits.go Populate byteMatchSz from caseFoldingEqualsASCII Mar 12, 2018
bits_test.go Use galloping search for nextFileIndex Feb 7, 2018
build-deploy.sh Add IndexMetadata.Version, populated by the link step May 17, 2018
contentprovider.go Incorporate shard rank into score Feb 9, 2018
eval.go Prefer matches from earlier documents Mar 5, 2018
hititer.go Handle the unlikely case of overflow for distanceHitIterator.next() Feb 6, 2018
index_test.go Don't abort indexing for UTF-8 errors May 25, 2018
indexbuilder.go Check for binary content using bytes.Index before building postings May 25, 2018
indexdata.go Populate byteMatchSz from caseFoldingEqualsASCII Mar 12, 2018
indexfile.go Unmap index files on close. Close index files early. Oct 20, 2016
indexfile_linux.go Fix infinite recursion in indexfile_linux.go. Nov 7, 2016
matchiter.go Populate byteMatchSz from caseFoldingEqualsASCII Mar 12, 2018
matchtree.go Drop coversContent logic Mar 12, 2018
read.go Copy content before returning results from the sharded searcher Mar 8, 2018
read_test.go Rationalize some file names May 5, 2017
section.go Use 64 bits for branch masks. May 8, 2017
section_test.go Convert postings to varint delta encoding on generation. Nov 10, 2016
toc.go Bump FeatureVersion due to SkipReason changes May 25, 2018
write.go Add IndexMetadata.Version, populated by the link step May 17, 2018

README.md

"Zoekt, en gij zult spinazie eten" - Jan Eertink

("seek, and ye shall eat spinach" - My primary school teacher)

This is a fast text search engine, intended for use with source code. (Pronunciation: roughly as you would pronounce "zooked" in English)

INSTRUCTIONS

Downloading:

go get github.com/google/zoekt/

Indexing:

go install github.com/google/zoekt/cmd/zoekt-index
$GOPATH/bin/zoekt-index .

Searching

go install github.com/google/zoekt/cmd/zoekt
$GOPATH/bin/zoekt 'ngram f:READ'

Indexing git repositories:

go install github.com/google/zoekt/cmd/zoekt-git-index
$GOPATH/bin/zoekt-git-index -branches master,stable-1.4 -prefix origin/ .

Indexing repo repositories:

go install github.com/google/zoekt/cmd/zoekt-{repo-index,mirror-gitiles}
zoekt-mirror-gitiles -dest ~/repos/ https://gfiber.googlesource.com
zoekt-repo-index \
   -name gfiber \
   -base_url https://gfiber.googlesource.com/ \
   -manifest_repo ~/repos/gfiber.googlesource.com/manifests.git \
   -repo_cache ~/repos \
   -manifest_rev_prefix=refs/heads/ --rev_prefix= \
   master:default_unrestricted.xml

Starting the web interface

go install github.com/google/zoekt/cmd/zoekt-webserver
$GOPATH/bin/zoekt-webserver -listen :6070

A more organized installation on a Linux server should use a systemd unit file, eg.

[Unit]
Description=zoekt webserver

[Service]
ExecStart=/zoekt/bin/zoekt-webserver -index /zoekt/index -listen :443  --ssl_cert /zoekt/etc/cert.pem   --ssl_key /zoekt/etc/key.pem
Restart=always

[Install]
WantedBy=default.target

SEARCH SERVICE

Zoekt comes with a small service management program:

go install github.com/google/zoekt/cmd/zoekt-indexserver

cat << EOF > config.json
[{"GithubUser": "username"},
 {"GitilesURL": "https://gerrit.googlesource.com", Name: "zoekt" }
]
EOF

$GOPATH/bin/zoekt-server -mirror_config config.json

This will mirror all repos under 'github.com/username' as well as the 'zoekt' repository. It will index the repositories.

It takes care of fetching and indexing new data and cleaning up logfiles.

The webserver can be started from a standard service management framework, such as systemd.

SYMBOL SEARCH

It is recommended to install Universal ctags to improve ranking. See here for more information.

ACKNOWLEDGEMENTS

Thanks to Alexander Neubeck for coming up with this idea, and helping me flesh it out.

DISCLAIMER

This is not an official Google product