Merge pull request #228 from ivg/final-fixes

Cleaner error handling and some bug fixes.

CHANGES.md

@@ -1,3 +1,72 @@
+0.9.8
+=====
+
+1. BAP IR is introduced
+
+   BAP Intermediate Representation is based on BAP Instruction
+   Language and is a semigraphical representation of a program.
+
+   See documentation and following PR's for more information.
+   a2a4621df7c5b25d85c04665732423992e8def98
+   74cdee48818225e8b43d39803c97471903ef6d1f
+
+2. Refactored structure of the Project
+   Module `Project` now a proper entry point to the library.
+   Many stuff from bap utility moved there.
+   See 96bd334a0d8af17a6dfd21eff9ec710d448f13e8 for more details.
+
+   This is a breaking change. It hides `project` record and removes
+   access to some information, that was previously marked as deprecated:
+   - symbols as a mapping from memory to string
+   - base as a memory.
+
+   Instead of old symbols table we now have a better interface, see
+   below. Instead of base, we now represent all memory as an interval
+   map (Memmap).
+
+
+3. New model for symbols
+
+   Previosly symbols were modeled as contiguous chunk of memory,
+   marked with name. Moreover, data sharing between different symbols
+   weren't allowed. Since this release, symbols can be a noncontiguous,
+   and share data. A new interface is implemented in `Symtab` module.
+
+4. Plugins dependency and autoloading
+
+   Plugins now can now specify dependencies to other plugins, that may
+   be auto-loaded by the library.
+   See db2a175ba8e6708753a06a2428940c857a1910ec
+
+5. Extended BIL helpers
+   See 65f472c08d27020a6570b7992b93397346251d1e
+
+6. Exposed ELF library
+
+7. Fixed segment/section/region name hell
+   See 9a574498392c6a13606c9d202037daf137bb780c
+
+8. New universal values library
+
+   The library is based on Core_kernel's Univ, but with addition of
+   serialization, comparison and pretty-printing.
+   See 383003d60baa3434dd4cd8c894e1d8c2e889b4a2
+
+9. Added bap-fsi-benchmark utility
+
+   80382114f395bcf45925ae2e4bc5b9aac5bba4e7
+
+10. Fixed BIL piqi serialization
+
+   2a5c4671468c5a2699b6007a8af3fda8867e8eb8
+
+11. Fixed installation on more recent ubuntu
+
+  By defaulting LLVM version to 3.4 (and more clever
+  searching procedure)
+
+12. Lot's of bugfixes and small extensions
+
 0.9.7
 =====
 

README.md

@@ -1,9 +1,9 @@
 # Overview
 
-[![Join the chat at https://gitter.im/BinaryAnalysisPlatform/bap](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/BinaryAnalysisPlatform/bap?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [![Build Status](https://travis-ci.org/BinaryAnalysisPlatform/bap.svg?branch=master)](https://travis-ci.org/BinaryAnalysisPlatform/bap) [![docs](https://img.shields.io/badge/doc-v0.9.7-green.svg)](http://binaryanalysisplatform.github.io/bap/api/v0.9.7/Bap.Std.html) [![docs](https://img.shields.io/badge/doc-master-green.svg)](http://binaryanalysisplatform.github.io/bap/api/master/Bap.Std.html)
+[![Join the chat at https://gitter.im/BinaryAnalysisPlatform/bap](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/BinaryAnalysisPlatform/bap?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) [![Build Status](https://travis-ci.org/BinaryAnalysisPlatform/bap.svg?branch=master)](https://travis-ci.org/BinaryAnalysisPlatform/bap) [![docs](https://img.shields.io/badge/doc-v0.9.8-green.svg)](http://binaryanalysisplatform.github.io/bap/api/v0.9.8/Bap.Std.html) [![docs](https://img.shields.io/badge/doc-master-green.svg)](http://binaryanalysisplatform.github.io/bap/api/master/Bap.Std.html)
 
 BAP is a platform for binary analysis. It is written in OCaml, but can
-be used from other languages, for example, from Python.
+be used from other languages.
 
 # <a name="Installation"></a>Installation
 
@@ -27,30 +27,32 @@ $ pip install git+git://github.com/BinaryAnalysisPlatform/bap.git
 
 ## Using from OCaml
 
-There're two ways to use BAP. Compile your own application, and use
-BAP library, or write a plugin, that can still use the library, but
-will also get an access to decompiled binary. For the latter, write
-your plugin in OCaml using your
+There're two ways to use BAP: compile your own stand-alone
+application, and use BAP library, or write a plugin, that can still
+use the library, but will also get an access to decompiled binary, as
+well as intergration with tools and other plugins. For the latter,
+write your plugin in OCaml using your
 [favorite text editor](https://github.com/BinaryAnalysisPlatform/bap/wiki/Emacs)
 :
+
 ```sh
-$ cat mycode.ml
-open Bap.Std
-let main project = print_endline "Hello, World"
-let () = Project.register_plugin' main
+$ cat hello_world.ml
+open Bap.Std let main project =
+print_endline "Hello, World"
+let () = Project.register_pass "hello-world" main
 ```
 
 Next, build it with our `bapbuild` tool:
 
 ```sh
-$ bapbuild mycode.plugin
+$ bapbuild hello_world.plugin
 ```
 
 After this you can load your plugin with `-l` command line option, and
 get an immediate access to the decompiled binary:
 
 ```sh
-$ bap /bin/ls -lmycode
+$ bap /bin/ls -lhello-world
 ```
 
 `bapbuild` can compile a standalone applications, not only plugins. In

_oasis

@@ -1,6 +1,6 @@
 OASISFormat: 0.4
 Name:        bap
-Version:     0.9.7
+Version:     0.9.8
 Synopsis:    BAP Core Library
 Authors:     BAP Team
 Maintainers: Ivan Gotovchits <ivg@ieee.org>

lib/bap/bap.mli

@@ -445,7 +445,17 @@ module Std : sig
 
   *)
 
-  (** {2:plugins Writing Program Analysis Plugins}
+  (** {2:project Working with project}
+
+      There're two general approaches to obtain a value of type
+      {{!Project}project}:
+      - create it manually using one of the [Project.from_*] function;
+      - to write a plugin to a [bap] utility
+
+      Although the first approach is simplistic and gives you a full
+      control, we still recommend to use the latter, as [bap] utility
+      will provide you integration with different tools, like IDA, as
+      well as interaction with a user and other plugins.
 
       To write a program analysis plugin (or pass in short) you need to
       implement a function with one of the following interfaces:
@@ -467,16 +477,32 @@ module Std : sig
       annotations, discover new symbols, make corrections, and even
       change the architecture and re-disassemble everything.
 
+      {3 Example}
+
+      The following plugin prints all sections in a file:
+
+      {[
+        open Core_kernel.Std
+        open Bap.Std
+        open Format
+
+        let print_sections p =
+          Project.memory p |> Memmap.to_sequence |> Seq.iter ~f:(fun (mem,x) ->
+              Option.iter (Value.get Image.section x) ~f:(fun name ->
+                  printf "Section: %s@.%a@." name Memory.pp mem))
+
+        let () = Project.register_pass' "print-sections" print_sections
+      ]}
+
       {3 Exchanging information}
 
       For exchanging information in a type safe manner, we use
       {{!Value}universal values}. Values can be attached to a
-      particular memory region, or put into the [storage]
+      particular memory region, IR terms, or put into the [storage]
       dictionary. For the first case we use the {{!Memmap}memmap} data
       structure.  It is an interval tree containing all the memory
       regions that are used during analysis. For the [storage] we use
-      [Dict] data structure. One can also annotate program by
-      attaching attributes to IR terms.
+      [Dict] data structure.
 
       {3 Memory marks}
 
@@ -1698,7 +1724,7 @@ module Std : sig
     val iter : unit #visitor -> stmt list -> unit
 
     (** [map mapper bil] map or transform BIL program. See also
-        {!Exp.map} and {!Stmt.map}. *)
+        {!Exp.map}. *)
     val map : #mapper -> stmt list -> stmt list
 
     (** [find finder bil] search in [bil] using provided [finder]. See
@@ -1834,20 +1860,20 @@ module Std : sig
         {!Bil.normalize_negatives} for more details  *)
     val normalize_negatives : t -> t
 
-    (** [fold_constants] performs one step of constant evaluation. In
+    (** [fold_consts] performs one step of constant evaluation. In
         order to perform all possible reductions one should use
         {!fixpoint} function, provided later. Example:
         [let x = Bil.var (Var.create "x" reg32_t)]
-        [fixpoint fold_constants Bil.(x lxor x lxor x lxor x)]
+        [fixpoint fold_consts Bil.(x lxor x lxor x lxor x)]
 
         will yield [0x0:32], but without
         a fixpoint, the result would be just:
 
         [fold_constants Bil.(x lxor x lxor x lxor x)]
         [(0x0:32 ^ x) ^ x].
 
-        See also {!Bil.fold_constants} and {!Stmt.fold_constants} *)
-    val fold_constants : t -> t
+        See also {!Bil.fold_consts} *)
+    val fold_consts : t -> t
 
     (** [fixpoint f] applies transformation [f] to [t] until it
         reaches a fixpoint, i.e., such point [x] that
@@ -3447,7 +3473,9 @@ module Std : sig
     (** Memory  *)
     val mem : var
 
-    (** Program counter  *)
+    (** Program counter.
+        @deprecated this maybe removed, if we decide, that PC register
+        breaks abstraction.*)
     val pc  : var
 
     (** Stack pointer  *)
@@ -3459,6 +3487,13 @@ module Std : sig
     val vf  : var
     val nf  : var
 
+    (** [addr_of_pc mem] given a memory region, representing some
+        instruction, return a value to which a PC register is set when
+        this instruction is executed by CPU.
+        This function is useful to resolve PC-relative calculations.
+        @deprecated this maybe removed, if we decide, that PC register
+        breaks abstraction.
+    *)
     val addr_of_pc : mem -> addr
 
     (** {3 Predicates}  *)
@@ -4035,7 +4070,7 @@ module Std : sig
       abstracts target architecture. The returned module has type
       {!Target} and can be unpacked locally with:
       {[
-        let module Target = (val Project.target project) in
+        let module Target = (val target_of_arch arch) in
       ]}
   *)
   val target_of_arch : arch -> (module Target)
@@ -5086,7 +5121,7 @@ module Std : sig
   module Project : sig
     (** A project groups together all the information recovered from
         the underlying binary. It is also used for exchanging
-        information between {{!plugins}passes}.  *)
+        information between {{!section:project}passes}.  *)
 
     type t
 
@@ -5236,6 +5271,44 @@ module Std : sig
 
     type 'a register = ?deps:string list -> string -> 'a -> unit
 
+    (** An error that can occur when loading or running pass.
+        - [Not_loaded name] pass with a given [name] wasn't loaded for
+          some reason. This is a very unlikely error, indicating
+          either a logic error in the plugin system implementation or
+          something very weird, that we didn't expect.
+
+        - [Is_duplicate name] more than one plugin were registered
+          under this [name], either it is the same plugin or a name clash
+          between to different we don't know.
+
+        - [Not_found name] when we tried to load plugin with a given
+          [name] we failed to find it in our search paths.
+
+        - [Doesn't_register name] the plugin with a given [name]
+          doesn't register a pass with the same name.
+
+        - [Load_failed (name,problem)] a [problem] has occured, when
+          we tried to dynamically link a plugin with a given [name]
+
+        - [Runtime_error (name,exn)] when plugin with a given [name]
+          was ran it raised an [exn].
+
+    *)
+    type error =
+      | Not_loaded of string
+      | Is_duplicate of string
+      | Not_found of string
+      | Doesn't_register of string
+      | Load_failed of string * Error.t
+      | Runtime_error of string * exn
+    with sexp_of
+
+    (** raised when a pass failed to load or to run. Note: this
+        exception is raised only from two functions in this module, that
+        state this in their documentation and has [_exn] suffix in their
+        name. *)
+    exception Pass_failed of error with sexp
+
     (** [register_pass_with_args name pass] registers [pass] that
         requires command line arguments. The arguments will be passed
         in the first parameter of the [pass] function.
@@ -5256,7 +5329,6 @@ module Std : sig
         (See {!register_pass_with_args})  *)
     val register_pass': (t -> unit) register
 
-
     (** [register_pass_with_args' name pass] register a [pass] that
         requires arguments for a side effect.
         (See {!register_pass_with_args}) *)
@@ -5292,6 +5364,27 @@ module Std : sig
          supported [--plugin-name-argument-name value].
     *)
     val run_passes : ?library:string list -> ?argv:string array -> t -> t Or_error.t
+
+
+    (** [passes ?library ()] returns a transitive closure of all
+        passes registered in the system so far.   *)
+    val passes : ?library:string list -> unit -> string list Or_error.t
+
+
+    (** [run_passes_exn proj] is the same as {!run_passes}, but raises
+        an exception on error. Useful to provide custom error
+        handling/printing.
+        @raise Pass_failed if failed to load, or if plugin failed at runtime.
+    *)
+    val run_passes_exn : ?library:string list -> ?argv:string array -> t -> t
+
+
+    (** [passes_exn proj] is the same as {!passes}, but raises
+        an exception on error. Useful to provide custom error
+        handling/printing.
+        @raise Pass_failed if failed to load some plugin *)
+    val passes_exn : ?library:string list -> unit -> string list
+
   end
 
   type project = Project.t