New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dependency Build Redesign for version 1.8.0 #2253

Closed
theopolis opened this Issue Jul 16, 2016 · 6 comments

Comments

Projects
None yet
2 participants
@theopolis
Contributor

theopolis commented Jul 16, 2016

Build change TL;DR

$ make distclean
$ make deps
$ make

Overview of build changes and assumptions

  1. On Linux and OS X the directory /usr/local/osquery is used as a virtual system root.
  2. All Linux builds statically compile libstdc++.
  3. OS X no longer uses /usr/local/ or supports your default or non-default brew install.
  4. CMake no longer searches for or uses BOOST_ROOT for boost dependencies.
  5. The boost -mt libraries are used.
  6. rpm_packages and rpm_package_files have been temporarily blacklisted, tracked in #2302.

FAQs

WHY!?
See the next section.

What is the deal with /usr/local/osquery?
This is a hard-coded directory the build system wants to install a Homebrew or Linuxbrew into.

How much disk space is this using?
On OS X: 450M, on Linux 1.5G

How large are the resultant binaries: osqueryi, osqueryd?
Linux unstripped: 50M
Linux stripped: 15M
OS X unstripped: 11M
OS X stripped: 8.3M

What system dependencies are needed to bootstrap dependency installs?
Mainly: make, python and ruby
If you want to build the dependencies a gcc 4.2+ or equivalent is needed.

What system dependencies are needed by the resultant binaries?
libgcc_s 4.2, libc 2.10, libz (but maybe for not much longer)

Can I compile dependencies myself?
See the section below about new build controls, but yes of course.

I use Linuxbrew/Homebrew and see SHA256 mismatch?
If you see Error: SHA256 mismatch when downloading dependencies, you need to clean your download cache:
OS X: rm ./Library/Cache/Homebrew/*.bottle.tar.gz
Linux: rm ~/.cache/Homebrew/*.bottle.tar.gz

make deps asks for sudo, for what?
Each time sudo is used, the requested command should be displayed directly above.
This is used during the dependency install to (1) install several system packages with your package manager, (2) create and edit permissions for /usr/local/osquery.

New build controls

These will soon be reflected in the build documentation. Each is an environment variable that should be set before make or make deps.

OSQUERY_BUILD_DEPS=1: instead of installing dependencies, this will attempt to compile as much as possible. Additional binaries will be installed into /usr/local/osquery and 2 C runtimes, and 2 compiles (gcc/llvm) will be built. Running make depsclean beforehand is recommended.

OSQUERY_BUILD_SHARED=1: this will "also" produce a libosquery.so and libosquery_additional.so for your convenience. The resultant osqueryi and osqueryd will be dynamically linked against these shared libraries.

SKIP_TABLES=1: skip the table compile build step.

OSQUERY_DEPS=/some/path: use an alternate path for the root. Use this with OSQUERY_BUILD_DEPS when running make deps and also supply it to make.

OSQUERY_BUILD_BOTTLES=1: produce binary bottles (.tar.gz) from each dependency locally compiled.

Dependency Build Problem

Please see #2169 for the original problem statement. To recap, the current make deps is weird and has become unmanageable for contributors and dependency upgrades.

What are osquery's dependencies

It really 'depends', haha, since osquery is supported on a large number of Linux distributions with birthdates in the early 2012s as well as official support for OS X 10.9-10.11 and upcoming support for Windows 10. But let us attempt to enumerate them!

Core dependencies

The list of "core" dependencies are more like "basic" dependencies but also define the requirements for the osquery SDK. The SDK is a less-often-talked-about feature set but essentially implements everything in osquery that is NOT a plugin.

  1. libc and a GLIBC runtime, preferably with support for version 2.16 of the ABI and beyond.
  2. A C++11 implementation, LLVM's for Darwin platforms and GCC's for Linux platforms.
  3. pthreads, dl, libm (math), libz, librt which are generally provided by the OS. These are not direct requirements, but rather used indirectly 90% of the time.
  4. A newer boost version, 1.58+ to satisfy other dependencies. A boost version in general providing system, filesystem and a regex implementation. Some builds will require icuuc for unicode support.
  5. Google Flags for argument parsing.
  6. Google Logging for the default logging facility and the logging plugin "routing/sink" APIs.
  7. Apache Thrift for the extension APIs. Extensions allow Facebook and other users to implement custom tables and plugins in C++, Python, Ruby, PHP, etc.
  8. LZ4 for stream compression, an optional feature for several logging plugins and RocksDB.

Additional, or extended, dependencies

The "extended" set of dependencies are used to build a libosquery_additional.a intermediate library. They are used in several plugins and osquery tables.

  1. RocksDB, Snappy, LZMA, Bzip2, to provide the default backing storage for events and persistent storage for the osquery daemon.
  2. OpenSSL for hash functions and a TLS implementation used for several config, logging, and other plugins.
  3. CPP-Netlib for the HTTP client used in the above plugins.
  4. Amazon's AWS SDK for several logger plugins.
  5. ncurses and readline for the osquery shell.
  6. Google Mock and Google Test for our C and C++ unit tests.

The large set of tables mostly relies on C and C++ parsing and data retrieval libraires.

There are some generic/shared dependencies on all platforms:

  1. Magic for file magic/header parsing and identification.
  2. libresolv for DNS resolvers.
  3. The Sleuth Kit for disk/block/file details. This also provides parsing libraries for the most popular file systems.
  4. YARA for more file and signature checking.
  5. libarchive is used frequently on OS X for parsing caches and other OS facility storage. Consider PKG parsing and browser extension parsing.

On Darwin the following frameworks are needed:

  1. CoreFoundation for lots of goodies.
  2. Security for Keychain access, ACL parsing, and additional binary signature features.
  3. OpenDirectory for the user and group APIs.
  4. DiskArbitration for disk details and disk events.
  5. FSEvents for file system details and file events.

On Linux the list is much larger:

  1. GCrypt and libgpg-error are needed for CryptSetup, used to parse disk details related to encryption settings.
  2. libdevmapper is also needed for CryptSetup and other disk-related details.
  3. libblkid again for disk parsing.
  4. libuuid for exactly that!
  5. libip4c for iptables introspection.
  6. libaudit for the Linux audit message parsing and Netlink socket management.
  7. libudev for device details and device events.
  8. Inotify for file events.

Debian-based distributions need:

  1. libapt-pkg and libdpkg for package parsing.

And RedHat-based distributions need:

  1. librpm and librpmio, which depend on NSS, NSPR, popt and optionally libselinux.

And then we "vendor-in" a bleeding edge of SQLite because we often request new features specifically for osquery, and want them available prior to formal releases.

  1. SQLite!

Tooling dependencies

The tooling around building, testing, and creating deployable packages is often just as difficult to setup as the code dependencies.

  1. For integration (end-to-end) testing, osquery uses Python's psutils and pytest.
  2. Tables use a spec DSO, which generates C++ plugins and the SQL schema, this is facilitated by some Python tooling and jinja2 templates.
  3. Packages are built using a Ruby tool called fpm.
  4. clang-format is used to manage code style.
  5. cpp-check and LLVM's sanitization frameworks provide static and dynamic analysis.
  6. The OS X Leaks tool and valgrind provide memory leak detection.
  7. CMake provides most of the build infrastructure.
  8. git is used for version detection, as well as submodule management for SQLite and GTest.

Why does osquery need to control dependencies

It doesn't really, but it is a very unfortunate experience to find and install all of these dependencies. Using a series of scripts we can make this effort simpler and provide some environment reproducibility.

  1. We use custom continuous build and test infrastructure, which requires a script for setting up new nodes.
  2. Deploying to an array of hosts requires relaxed assumptions about the installed libraries and tools. Since most of the dependencies are not installed 'by default' we must link them statically.
  3. Debugging compile and runtime errors is much easier if the build environment is reproducible.
  4. Most of the dependencies are not available at the version we need, or at all.
  5. Various security features require explicit control over the compiler and compile flags used to build the static libraries.
  6. Tooling around code sanitization, runtime performance testing, style and formatting require newer compilers and work better having compiled most of the statically linked code.

The current dependency management solution

The Makefile provides make deps that does a few things:

  1. Detects your platform: a combination of what we call OS and DISTRO. The OS will be something like "darwin", "ubuntu", "freebsd", "centos", "rhel", or "windows". The DISTRO will be the version.
  2. Runs a provision script that installs a series of packages using the OS-defined package management solution, in some cases this is 120+ new packages.
  3. Runs a few additional "download", "unpack", "comple", "install" routines for dependencies that are exotic to the OS/DISTRO.

All of this runs several commands with sudo, and collides with anything you may have installed into /usr/local. On Linux the potentially-conflicting PREFIX is used to mimic Homebrew's choice on OS X.

In concert, the make callin to CMake assumes the host has run make deps and sets up include, link, and other search paths to use the /usr/local PREFIX.

The current dependency management issues

To recap again, this current solution has several limitations.

  1. It does not provide version management for all dependencies. Exotic dependencies are not tracked and are difficult to upgrade. There is no tooling to upgrade or alert the build host that a dependency's version was changed. Newer builds will fail for unknown reasons.
  2. Not 'all' OS/DISTRO combinations are represented in the osquery continuous build. This causes a drift in support for those combinations.
  3. Supporting 4 years worth of dependencies on distributions leads to an unwieldy number of edge cases for package versions (lots of if statements in bash).
  4. The continuous build cannot "boot test" from base install as it takes over an hour in most cases.
  5. Package managers may not respect install-time build options when upgrading (e.g., Homebrew). To update a package the existing one must be uninstalled.
  6. Static libraries provided by package managers, think -dev, -devel, or --with-static provide no guarantees over compiler or compile options. These libraries may be built with various levels of optimization, IA extension optimizations, and security features.

Summary of desired features

  1. A reproducible and more universal definition of required dependencies.
  2. A method to "install from cache" to improve build speeds.
  3. Control of compile flags for all C++ libraries.
  4. Control over Python, CMake, Ruby, and the Compiler version.
  5. Contributor 'testability' for new dependencies or dependency upgrades.
  6. Less build hosts required to target more OS and DISTRO flavors.

The solution: A Build Dependency Redesign!

We propose a 'new' way of establishing a build environment that focuses on supporting two modes of bootstrapping almost everything any OS needs.

This uses Homebrew and Linuxbrew plus an "in-repo" Homebrew Tap defining all of the build tools and build dependencies. make deps then clones a new isolated Homebrew or Linuxbrew, installs our Tap, and follows a OS X or Linux unified flow for bootstrapping dependencies.

That flow is more or less:

  1. Install some duplicate basic build tools into a custom prefix.
  2. Download or build a modern version of GCC: 5.3 for example.
  3. Download or build curl, Python, CMake, and other C dependencies with GCC.
  4. Download or build LLVM and Clang 3.6 using the modern GCC and installed tools.
  5. Download or build the remaining C++ dependencies using Clang.
  6. And finally, build osquery using Clang.

These steps include a "download" or "build" because this new solution allows anyone to replicate the build environment from almost scratch. Alternatively, and more preferably for most, Homebrew can bottle and allow us to distribute the built output.

With the "in-repo" Tap, we allow all osquery contributors to edit dependency Formulas alongside code changes. Homebrew manages versions and each Formula defines osquery-specific compile flags.

What changes

  1. The resultant osquery binaries become larger, from 10M to about 15M on Linux.
  2. New dependencies will require changes to ./tools/provision/formulas/.
  3. Although it can be overridden with environment variables, /usr/local/osquery is the reserved build and PREFIX location.

Implementation details

Each distribution defines a distro_main shell function within ./tools/provision/DISTRO.sh that updates the host and installs a very minimal set of packages:

  1. Git
  2. Ruby
  3. GCC (any version)
  4. wget
  5. Bzip2
  6. unzip

These are more-or-less the requirements for brew and to bootstrap a modern GCC compiler. In the case of Bzip2 and other packages that include shared or static C libraries, these are inconsequential and will be ignored by the osquery build.

The latest Homebrew or Linuxbrew is installed to '/usr/local/osquery. A symlink to./tools/provision/formulasis installed as the **osquery/homebrew-osquery-local** Tap. A unified bootstrap function is called, defined in./tools/provision.sh` that operates in one of two modes depending on environment flags.

Default mode: The script ignores package installs that are required to compile dependencies. The script unpacks GCC, LLVM, CMake, and other tools needed to compile osquery and unpacks the static library dependencies.

Build mode: The script unpacks basic tools needed to compile GCC 5.3 using a GCC provided by the OS. LLVM and CMake are compiled from source, along with about 90% of osquery's dependencies.

The Build mode output can be bottled and used for subsequent Default modes. This is exactly how the default dependency setup works. An initial build mode was run and the resultant bottles are uploaded to osquery's S3 bucket osquery-packages. Changes to dependencies within Pull Requests can trigger a rebuild of a single package by bumping the package version or requested bottle revision. This is documented within the Homebrew Formula and Bottle DSOs.

Immediate needs

This new build architecture unblocks immediate needs for new dependencies:

  1. A readline replacement called linenoise-ng. This will be the standard across Windows, OS X, and Linux.
  2. Configuration parsing using Argus.
  3. Updates to Amazon's AWS SDK.
  4. Updates to RocksDB

Future needs and benefits

The redesign allows two cool things today, a complete PIE osqueryi and osqueryd as well as the ability to build and distribute a libosquery.so.

Further refinement of the architecture will allow osquery to report the versions/hashes of dependencies it was built with and support a more-complete 'repeatable build'. It will soon be possible to build osquery in two different places and compare the deterministic output.

@theopolis

This comment has been minimized.

Contributor

theopolis commented Jul 16, 2016

Controlling the new make deps

To enable the build mode define the environment variable OSQUERY_BUILD_DEPS:

$ OSQUERY_BUILD_DEPS=1 make deps

Testing the build multiple times, simply remove the osquery-reserved PREFIX:

$ make depsclean

To override the default PREFIX define the environment variable OSQUERY_DEPS [1]:

OSQUERY_DEPS=/tmp/osquery make deps

[1] Note that overriding the default PREFIX will cause a most packages to be compiled. It is best to overriding only when using OSQUERY_BUILD_DEPS.

@zwass

This comment has been minimized.

Contributor

zwass commented Jul 20, 2016

This is going to be really awesome. Thank you for doing it, Ted.

@zwass

This comment has been minimized.

Contributor

zwass commented Aug 1, 2016

The documentation is amazing, and I look forward to using this new build system.

@zwass

This comment has been minimized.

Contributor

zwass commented Aug 2, 2016

On OSX, I had to do rm ~/Library/Caches/Homebrew/*.bottle.tar.gz to resolve the SHA256 Mismatch error. Is it a typo in the docs above, or do we have different setups?

@theopolis

This comment has been minimized.

Contributor

theopolis commented Aug 2, 2016

Definitely a typo, thanks!

@theopolis

This comment has been minimized.

Contributor

theopolis commented Aug 10, 2016

Ubuntu (and other Linux): cannot find crti.o: No such file or directory

If you see:

    /usr/local/osquery/bin/ld: cannot find crti.o: No such file or directory
    collect2: error: ld returned 1 exit status
    error: command '/usr/local/osquery/bin/gcc' failed with exit status 1

For example the PR/build: https://jenkins.osquery.io/job/osqueryPullRequestBuild/TargetSystem=ubuntu16/2996/console

Fix: make depsclean and make deps.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment