Release Notes
Added
- A
--case-sensitive flag for the download and parse subcommands. The documentation now states that file extensions are always case-sensitive (#9, reported by @Sparcraps).
- A
--no-output (or --count) flag for the parse subcommand that allows users to skip writing the extracted functions to disk and only collect their statistics (#5, reported by @Michago6).
- A
--lambdas flag for the parse subcommand that allows users to choose whether to extract lambda functions as well. By default, lambda functions are not extracted (#3, reported by @
linusbrew).
- An
overlap similarity criterion for the duplicate_files subcommand, which detects near-miss duplicates, that is, files that differ by a few statements rather than matching exactly. Files are compared by how many tokens they have in common, using the prefix filtering technique of SourcererCC and its adaptive extension. The proportion of how many tokens two files must share to be considered duplicates can be set with the --threshold flag (defaults to 0.8).
The --prefix (or -p) flag sets how far the prefix used to reject candidates may be deepened. A deeper prefix rejects more candidates before the full comparison, at the cost of more index lookups. Defaults to 1. The --languages (or -l) maps file extensions to languages, in the same JSON format used by the other subcommands. The overlap criterion compares files only against others of the same language, and leaves files whose extension belongs to no listed language uncompared. The other criteria ignore the flag. (PR #2 by @swartling)
Changed
- The
download subcommand takes its number of threads with -n or --threads instead of as a value without a flag, and rejects it without --skip. The parse and duplicate_files subcommands also accept --threads.
Fixed
- An issue with the
--regex flag in the download and parse subcommands that added word delimiters to the regexes when the flag was used (reported by @Smexykex).
- A parsing error in the
download subcommand when the --sub flag was used (#6, reported by @Michago6).
- An issue with the
download subcommand that caused it to not resume progress when restarted (#7, reported by @Smexykex).
- An issue with the
download subcommand that caused it to match extension prefixes instead of the full name (#8, reported by @Smexykex).
- Resuming: the
languages and metadata subcommands read the input file instead of the output file and processed nothing, pull_request failed when --ids was not id, and ids failed on an output file with only a header.
- Requests to the GitHub API (
ids, languages, metadata, pull_request): the final status after redirects is used, rate limits, server errors and network errors are retried instead of being written as error rows, % is no longer doubled, and nothing is printed to the standard output.
- Error rows of the
languages and metadata subcommands no longer break the CSV format when the message contains a comma, and start with http/ or error:. The filter_languages and filter_metadata subcommands discard both, instead of only HTTP/2 errors.
- The
download subcommand retries failed downloads, stops all threads after an error instead of working on without logging, no longer hangs when a thread panics, and logs archives that cannot be extracted as error rows. Re-extracting over symbolic links and paths that are not valid UTF-8 no longer stop the run.
- Keyword matching (
download, parse): keywords such as c++ or #include now match as whole words, the longer of two overlapping keywords (long double, long) is always preferred, and an inline regex flag only applies to its own keyword. Results of earlier runs can differ. The Ada keyword ** of the example file fp_others.json is fixed as well.
- The
ids subcommand stops at --max or at the most recent repository in linear mode, and reports --min not smaller than --max instead of crashing.
- The
filter_metadata subcommand no longer discards repositories whose last push is earlier than their creation date.
- The
languages subcommand lists the languages of a repository in the same order in every run.
- The
forks subcommand reports entries without a fork value instead of counting them as forks.
- The
duplicate_files subcommand stops after the first error and requires at least one thread. With several threads, the file standing for a group of duplicates is now always its first file in the input instead of changing between runs. The duplicates map is no longer overwritten without --force, a path listed several times in the input counts as one file, and a count column of the input is replaced instead of producing count_right.
- The
parse subcommand no longer hangs with 0 threads, skips files in languages it cannot parse instead of stopping the run (for example .h files mapped to c_header), and applies the --failures policies as described: skip-file and skip-function write rows with -1 for every statistic, and abort stops all threads.
- More precise statistics in the output of the
parse subcommand: a few missing node kinds were added (for example do and foreach loops), comments are now removed in Rust, and parameters, constructors and C++ templates are handled more accurately.
- Input files are read by column name instead of position, and tokens are read from the
token column instead of the first one.
- Empty values in id, name or token columns, and values that are not strings in keywords files, are reported as errors instead of being read as 0 or an empty string, or causing a crash.
- Log messages are printed when the standard error is not a terminal, for example in a batch job.
- Output files are flushed, so that write errors, for example on a full disk, are reported.
- Paths with mixed separators, such as
.\projects\/0/, in the outputs (#10, reported by @Sparcraps).
- The
--cache flag of the languages and metadata subcommands could assign the row of a project to another one.
- Reading a file no longer creates its parent directories, and creating a directory where a file exists is reported as an error.
Removed
- The
extract_benchmarks subcommand has been removed as it was too brittle.
Install scyros 0.4.0
Install prebuilt binaries via shell script
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/fxpl/scyros/releases/download/v0.4.0/scyros-installer.sh | sh
Install prebuilt binaries via powershell script
powershell -ExecutionPolicy Bypass -c "irm https://github.com/fxpl/scyros/releases/download/v0.4.0/scyros-installer.ps1 | iex"
Download scyros 0.4.0
