Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Framework, scripts and tests. #10

Closed
wants to merge 4 commits into from

Conversation

Projects
None yet
4 participants
@isle2983
Copy link

commented Mar 9, 2017

The bin/ directory holds scripts meant to be user-executable and helpful. It contains a collection of functionality that was previously submitted to contrib/devtools/, but as discussed, this repo is now considered the best place. These differ from the earlier submission in that they are now built off of the framework and have a bunch of extra amenities provided by the shared code.

The framework/ directory holds common infrastructure that can help accelerate future scripts and automation.

The test/ directory is a set of scripts that test the scripts in the bin/ directory and are invoked by .travis.yml to prevent breakage upon future refactoring and enhancement.

The scripts all have -h output courtesy of argparse that describe functionality and options. The bin/README.md file also gives a quick overview.

Out of the box, this can build from scratch and compute the full set of reports for the master branch of bitcoin/bitcoin like so:

$ bin/clone_configure_build.py ~/bitcoin-build
(output snipped)
$ bin/reports.py ~/bitcoin-build
(output snipped)

The BerkeleyDb 4.8 workspace and other miscellaneous file are written to /tmp/bitcoin-maintainer-tools/ (by default). For the clang toolchain binaries, it defaults to using the highest installed version in PATH, but it can also be pointed to a different toolchain directory (e.g. -b ~/clang+llvm-3.9.1-x86_64-linux-gnu-debian8/).

Where it makes sense, the scripts allow limiting the scope to individual files and/or subdirectories. Illustrated by example:

$ bin/clang_format.py check ~/bitcoin-build/src/init.cpp
--------------------------------------------------------------------------------
1472 files tracked in repo
 407 files in scope according to .bitcoin_maintainer_tools.json settings
   1 files examined according to listed targets
   4 parallel jobs for computing analysis
--------------------------------------------------------------------------------
A code format issue was detected in /home/isle/bitcoin-build/src/init.cpp

 +--------+         +------------+--------+---------+-----------+------------+
 | score  |         |     before |  added | removed | unchanged |      after |
 +--------+ +-------+------------+--------+---------+-----------+------------+
 | 90.39% | | lines |       1644 |    148 |     158 |      1486 |       1634 |
 +--------+ +-------+------------+--------+---------+-----------+------------+
--------------------------------------------------------------------------------
These files can be formatted by running:
$ clang_format.py format [option [option ...]] [target [target ...]]
--------------------------------------------------------------------------------

*** code format issue found

The repository is and .clang_format file is inferred from the targets. They can be formatted and re-checked like so:

$ bin/clang_format.py format ~/bitcoin-build/src/init.cpp
$ bin/clang_format.py check ~/bitcoin-build/src/init.cpp
--------------------------------------------------------------------------------
1472 files tracked in repo
 407 files in scope according to .bitcoin_maintainer_tools.json settings
   1 files examined according to listed targets
   4 parallel jobs for computing analysis
--------------------------------------------------------------------------------
No format issues found!
--------------------------------------------------------------------------------

Also, the report and check subcommands of the scripts all have a --json mode that provides the full information to assist with programmatic uses (bisecting, charting, etc.).

The intention is to provide utility for the main bitcoin/bitcoin project, but the scripts appear to work well for forks too. For example, you can point to Elements, build, and compute the analysis and reports:

$ bin/clone_configure_build.py -u https://github.com/ElementsProject/elements -b elements-0.13.1 ~/elements-build
$ bin/reports.py ~/elements-build

If the target repo provides a .bitcoin-maintainer-tools.json file, it helps the tools to know which subtrees to ignore, etc. If the repo doesn't have one, it uses .fallback-bitcoin-maintainer-tools.json which holds the appropriate settings for v0.14.0.

@JeremyRand JeremyRand referenced this pull request May 7, 2017

Open

Static analysis #108

@laanwj

This comment has been minimized.

Copy link
Member

commented Dec 13, 2017

Thanks, sorry, never noticed this PR before! That's a lot of new tools.
Will have a look.


class Checks(RepositoryCmds):
"""
Invokes several underlying RepositoryCmd check command instances and and

This comment has been minimized.

Copy link
@ChrisJBurns
class ClangFind(object):
"""
Assist finding clang tool binaries via either a parameter pointing to
a directory or by examinining the environment for installed binaries.

This comment has been minimized.

Copy link
@ChrisJBurns

ChrisJBurns Dec 13, 2017

examinining I'm guessing is supposed to be examining?


def finish_clang_settings(settings):
"""
Makes the settings namepsce uniform by instantiating classes filled in with

This comment has been minimized.

Copy link
@ChrisJBurns

ChrisJBurns Dec 13, 2017

namepsce spelling? Supposed to be namespace?

Wraps and utilizes regular expression to quickly evaluate whether a path
is included or excluded from a desired set of files. The desired set of
files to include or exclude is indicated by appending a list of fnmatch
expressions. The expressions are evaluated is the order that they are

This comment has been minimized.

Copy link
@ChrisJBurns

ChrisJBurns Dec 13, 2017

is supposed to be in?

@ChrisJBurns

This comment has been minimized.

Copy link

commented Dec 13, 2017

Some spelling/grammar errors noticed


TARBALL = "db-4.8.30.NC.tar.gz"
DOWNLOAD_URL = "http://download.oracle.com/berkeley-db/" + TARBALL
CHECKSUM = "12edc0df75bf9abd7f82f821795bcee50f42cb2e5f76a6a281b85732798364ef"
@MarcoFalke

This comment has been minimized.

Copy link
Contributor

commented Jan 3, 2018

Could make sense to split up in pulls for each tool separately to make review easier in small steps.

@@ -0,0 +1,545 @@
#!/usr/bin/env python3

This comment has been minimized.

Copy link
@MarcoFalke

MarcoFalke Aug 9, 2018

Contributor

Sorry for asking you to move this file here. I think we want to keep it in the core repo for the purpose of running it as a scripted-diff.

@MarcoFalke

This comment has been minimized.

Copy link
Contributor

commented Apr 29, 2019

Closing for now. Let me know when you want to work on this again

@MarcoFalke MarcoFalke closed this Apr 29, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.