Skip to content

Releases: tamada/pelicode

Release list

Release v0.1.0

Release v0.1.0 Latest
Latest

Choose a tag to compare

View all tags
@github-actions github-actions released this 19 Oct 01:08
v0.1.0
617dc6a
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.

The first release of pelicode, a research prototype that tells you where your code may have come from.

pelicode keeps a database of the functions of published open-source packages, and checks a project, or only what a change touches, against it. It reports similar code candidates to review, each with the package, version, file, lines and license it was found in. It does not decide whether code was copied: similar code is no proof of copying, and copying is not always a problem.

The database holds hashes, locations and licenses, never source code.

What it can do

  • Languages: Go, Java, Kotlin, Python, Ruby and Rust, function by function (Tree-sitter), normalized at four levels: layout, identifiers and literals, types, structure.
  • Repositories: PyPI, Maven Central, the Go module proxy, RubyGems and crates.io.
    • Downloads are verified against the digests the repositories publish, licenses are identified, and requests are paced.
    • crawl --top N fetches the best-known packages of each.
    • crawl --updates fetches the new versions of the packages in a database.
  • Matching: exact matches after normalization, and similar matches through winnowing fingerprints, MinHash and LSH. Code common to many packages is left out, and so are the dependencies of the checked project, read from its lock files.
  • Checking: directories, a git range (--git main...HEAD), or what is staged (--staged, for pre-commit hooks). The results come as text, JSON, CSV, or SARIF for GitHub code scanning.
  • Settings file: .pelicode.yaml or .pelicode.json5 in the repository.
  • Sharing databases:
    • pelicode snapshot writes a copy for checking only.
    • Snapshots of the best-known 500 packages of each ecosystem are published monthly in quay.io (quay.io/tama5/pelicode-db).
    • scan --db oci://quay.io/tama5/pelicode-db:pypi-latest fetches, verifies and caches one; several can be checked at once.
  • Maintenance: pelicode stats refreshes the counts of common hashes incrementally, and pelicode clean removes data no version refers to any more.

Install

brew install tamada/tap/pelicode
cargo install pelicode-cli
docker run --rm -v "$PWD":/workdir quay.io/tama5/pelicode --help

Archives for Linux, macOS and Windows (x64 and arm64) are attached to this release. See the installation guide and Getting Started.

pelicode scan ./src --db oci://quay.io/tama5/pelicode-db:pypi-latest

Known limitations

  • Defaults: the default parameters (k, thresholds, the stop threshold) rest on preliminary observations; the evaluation planned in §11 of the specification has not been done yet.
  • git: --git and --staged need the git command (#6).
  • Rankings: the best-known packages of Maven Central and Go are ranked by the number of repositories that depend on them, since those repositories publish no download counts.
  • Scale: measured up to 1.3 million functions per database (§11.3 of the specification); larger databases are untried.
  • GitHub Action: the action for pull requests lives in tamada/pelicode-action, with releases of its own.
Assets 8
Loading