Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Dependency Build Redesign for version 1.8.0 #2253
Build change TL;DR
Overview of build changes and assumptions
What is the deal with
How much disk space is this using?
How large are the resultant binaries:
What system dependencies are needed to bootstrap dependency installs?
What system dependencies are needed by the resultant binaries?
Can I compile dependencies myself?
I use Linuxbrew/Homebrew and see SHA256 mismatch?
New build controls
These will soon be reflected in the build documentation. Each is an environment variable that should be set before
Dependency Build Problem
Please see #2169 for the original problem statement. To recap, the current
What are osquery's dependencies
It really 'depends', haha, since osquery is supported on a large number of Linux distributions with birthdates in the early 2012s as well as official support for OS X 10.9-10.11 and upcoming support for Windows 10. But let us attempt to enumerate them!
The list of "core" dependencies are more like "basic" dependencies but also define the requirements for the osquery SDK. The SDK is a less-often-talked-about feature set but essentially implements everything in osquery that is NOT a plugin.
Additional, or extended, dependencies
The "extended" set of dependencies are used to build a
The large set of tables mostly relies on C and C++ parsing and data retrieval libraires.
There are some generic/shared dependencies on all platforms:
On Darwin the following frameworks are needed:
On Linux the list is much larger:
Debian-based distributions need:
And RedHat-based distributions need:
And then we "vendor-in" a bleeding edge of SQLite because we often request new features specifically for osquery, and want them available prior to formal releases.
The tooling around building, testing, and creating deployable packages is often just as difficult to setup as the code dependencies.
Why does osquery need to control dependencies
It doesn't really, but it is a very unfortunate experience to find and install all of these dependencies. Using a series of scripts we can make this effort simpler and provide some environment reproducibility.
The current dependency management solution
All of this runs several commands with
In concert, the
The current dependency management issues
To recap again, this current solution has several limitations.
Summary of desired features
The solution: A Build Dependency Redesign!
We propose a 'new' way of establishing a build environment that focuses on supporting two modes of bootstrapping almost everything any OS needs.
This uses Homebrew and Linuxbrew plus an "in-repo" Homebrew Tap defining all of the build tools and build dependencies.
That flow is more or less:
These steps include a "download" or "build" because this new solution allows anyone to replicate the build environment from almost scratch. Alternatively, and more preferably for most, Homebrew can bottle and allow us to distribute the built output.
With the "in-repo" Tap, we allow all osquery contributors to edit dependency Formulas alongside code changes. Homebrew manages versions and each Formula defines osquery-specific compile flags.
Each distribution defines a
These are more-or-less the requirements for
The latest Homebrew or Linuxbrew is installed to '/usr/local/osquery
Default mode: The script ignores package installs that are required to compile dependencies. The script unpacks GCC, LLVM, CMake, and other tools needed to compile osquery and unpacks the static library dependencies.
Build mode: The script unpacks basic tools needed to compile GCC 5.3 using a GCC provided by the OS. LLVM and CMake are compiled from source, along with about 90% of osquery's dependencies.
The Build mode output can be bottled and used for subsequent Default modes. This is exactly how the default dependency setup works. An initial build mode was run and the resultant bottles are uploaded to osquery's S3 bucket
This new build architecture unblocks immediate needs for new dependencies:
Future needs and benefits
The redesign allows two cool things today, a complete PIE
Further refinement of the architecture will allow osquery to report the versions/hashes of dependencies it was built with and support a more-complete 'repeatable build'. It will soon be possible to build osquery in two different places and compare the deterministic output.
Controlling the new
Ubuntu (and other Linux): cannot find crti.o: No such file or directory
If you see:
For example the PR/build: https://jenkins.osquery.io/job/osqueryPullRequestBuild/TargetSystem=ubuntu16/2996/console