Add binary support

Cargo.toml

@@ -1,7 +1,7 @@
 [package]
 authors = ["konstin <konstin@mailbox.org>"]
 name = "pyo3-pack"
-version = "0.2.0"
+version = "0.3.0"
 description = "Build and publish crates with pyo3 bindings as python packages"
 exclude = ["get-fourtytwo/**/*", "integration-test/**/*", "sysconfig/*"]
 readme = "Readme.md"
@@ -59,6 +59,7 @@ password-storage = ["upload", "keyring"]
 [workspace]
 members = [
     "get-fourtytwo",
+    "hello-world"
 ]
 
 # This is necessary due to https://github.com/eqrion/cbindgen/issues/203

Readme.md

@@ -6,7 +6,7 @@
 [![API Documentation on docs.rs](https://docs.rs/pyo3-pack/badge.svg)](https://docs.rs/pyo3-pack)
 [![Chat on Gitter](https://img.shields.io/gitter/room/nwjs/nw.js.svg?style=flat-square)](https://gitter.im/PyO3/Lobby)
 
-Build and publish crates with pyo3, rust-cpython or (c)ffi bindings as python packages.
+Build and publish crates with pyo3, rust-cpython and cffi bindings as well as rust binaries as python packages.
 
 This project was meant as a zero configuration replacement for [setuptools-rust](https://github.com/PyO3/setuptools-rust). It supports building wheels for python 2.7 and 3.5+ on windows, linux and mac and can upload them to [pypi](https://pypi.org/).
 
@@ -18,15 +18,21 @@ You can either download binaries from the [latest release](https://github.com/Py
 cargo install pyo3-pack
 ```
 
+You can also install pyo3-pack with pip:
+
+```shell
+pip install pyo3-pack
+```
+
 There are three main subsommands:
 
  * `publish` builds the crate into python packages and publishes them to pypi.
  * `build` builds the wheels and stores them in a folder (`target/wheels` by default), but doesn't upload them.
  * `develop` builds the crate and install it's as a python module directly in the current virtualenv
 
-pyo3-pack runs directly on a crate, with no extra files needed, and also doesn't clash with an existing setuptools-rust or milksnake configuration. You can even integrate it with testing tools such as tox (see `get-fourtytwo` for an example).
+`pyo3` and `rust-cpython` bindings are automatically detected, for cffi or binaries you need to pass `-b cffi` or `-b bin`. pyo3-pack needs no extra configuration files, and also doesn't clash with an existing setuptools-rust or milksnake configuration. You can even integrate it with testing tools such as tox (see `get-fourtytwo` for an example).
 
-The name of the package will be the name of the cargo project, i.e. the name field in the `[package]` section of Cargo.toml. The name of the module, which you are using when importing, will be the `name` value in the `[lib]` section (which defaults to the name of the package).
+The name of the package will be the name of the cargo project, i.e. the name field in the `[package]` section of Cargo.toml. The name of the module, which you are using when importing, will be the `name` value in the `[lib]` section (which defaults to the name of the package). For binaries it's simply the name of the binary generated by cargo.
 
 Pip allows adding so called console scripts, which are shell commands that execute some function in you program. You can add console scripts in a section `[package.metadata.pyo3-pack.scripts]`. The keys are the script names while the values are the path to the function in the format `some.module.path:class.function`, where the `class` part is optional. The function is called with no arguments. Example:
 
@@ -35,7 +41,7 @@ Pip allows adding so called console scripts, which are shell commands that execu
 get_42 = "get_fourtytwo:DummyClass.get_42"
 ```
 
-For pyo3 and rust-cpython, pyo3-pack can only build packages for installed python versions, so you might want to use e.g. pyenv, deadsnakes or docker for building. If you don't set your own interpreters with `-i`, a heuristic is used to search for python installations. You can get a list of those with the `list-python` subcommand. cffi wheels are compatible with all python versions, but they need to have `cffi` installed to build (`pip install cffi`).
+For pyo3 and rust-cpython, pyo3-pack can only build packages for installed python versions, so you might want to use pyenv, deadsnakes or docker for building. If you don't set your own interpreters with `-i`, a heuristic is used to search for python installations. You can get a list of those with the `list-python` subcommand. cffi wheels are compatible with all python versions, but they need to have `cffi` installed to build (`pip install cffi`).
 
 ### Build
 
@@ -107,14 +113,17 @@ OPTIONS:
 
 ```
 USAGE:
-    pyo3-pack develop [OPTIONS]
+    pyo3-pack develop [FLAGS] [OPTIONS]
 
 FLAGS:
     -h, --help       Prints help information
+        --release    Compile in release mode. This is useful e.g. for benchmarking
     -V, --version    Prints version information
 
 OPTIONS:
-    -b, --bindings-crate <binding_crate>            The crate providing the python bindings [default: pyo3]
+    -b, --bindings-crate <binding_crate>
+            The crate providing the python bindings. pyo3, rust-cpython and cffi are supported
+
         --cargo-extra-args <cargo_extra_args>...
             Extra arguments that will be passed to cargo as `cargo rustc [...] [arg1] [arg2] --`
 
@@ -123,16 +132,42 @@ OPTIONS:
             Extra arguments that will be passed to rustc as `cargo rustc [...] -- [arg1] [arg2]`
 ```
 
-### Manylinux and auditwheel
+
+## Cffi
+
+For crates with cffi you need to use a build script that writes c headers to a file called `target/header.h` until [eqrion/cbdingen#203](https://github.com/eqrion/cbindgen/issues/203) is resolved:
+
+```rust
+extern crate cbindgen;
+
+use std::env;
+use std::path::Path;
+
+fn main() {
+    let crate_dir = env::var("CARGO_MANIFEST_DIR").unwrap();
+
+    let mut config: cbindgen::Config = Default::default();
+    config.language = cbindgen::Language::C;
+    cbindgen::generate_with_config(&crate_dir, config)
+        .expect("Unable to generate bindings")
+        .write_to_file(Path::new("target").join("header.h"));
+}
+```
+
+## Manylinux and auditwheel
 
 For portability reasons, native python modules on linux must only dynamically link a set of very few libraries which are installed basically everywhere, hence the name manylinux. The pypa offers a special docker container and a tool called [auditwheel](https://github.com/pypa/auditwheel/) to ensure compliance with the [manylinux rules](https://www.python.org/dev/peps/pep-0513/#the-manylinux1-policy). pyo3-pack contains a reimplementation of the most important part of auditwheel that checks the generated library, so there's no need to use external tools. If you want to disable the manylinux compliance checks for some reason, use the `--skip-auditwheel` flag.
 
+Note that the pyo3-pack pip package is not manylinux compliant (A compliant package could neither upload not use the keyring)
+
 ## Code
 
 The main part is the pyo3-pack library, which is completely documented and should be well integratable. The accompanying `main.rs` takes care username and password for the pypi upload and otherwise calls into the library.
 
-There are the `get_fourtytwo` crate with pyo3 bindings and the `points` crate with (c)ffi bindings, which are used as examples and for integration testing. The `sysconfig` folder contains the output of `python -m sysconfig` for different python versions and platform, which is helpful during development.
+Without the `upload` and `password-storage` features, pyo3-pack itself is manylinux compliant (and has much less dependencies).
+
+There are three different examples, which are also used for integration testing: `get_fourtytwo` with pyo3 bindings, `points` crate with cffi bindings and `hello-world` as a binary. The `sysconfig` folder contains the output of `python -m sysconfig` for different python versions and platform, which is helpful during development.
 
-You need to install `virtualenv` (`pip install virtualenv`) to run the tests.
+You need to install `virtualenv` and `cffi` (`pip install virtualenv cffi`) to run the tests.
 
 You might want to have look into my [blog post](https://blog.schuetze.link/2018/07/21/a-dive-into-packaging-native-python-extensions.html) which explains the intricacies of building native python packages.

hello-world/Cargo.toml

@@ -0,0 +1,6 @@
+[package]
+name = "hello-world"
+version = "0.1.0"
+authors = ["konstin <konstin@mailbox.org>"]
+
+[dependencies]

hello-world/check_installed.py

@@ -0,0 +1,12 @@
+from subprocess import check_output
+
+
+def main():
+    output = check_output(["hello-world"]).decode("utf-8").strip()
+    if not output == "Hello, world!":
+        raise Exception(output)
+    print("SUCCESS")
+
+
+if __name__ == '__main__':
+    main()

hello-world/src/main.rs

@@ -0,0 +1,3 @@
+fn main() {
+    println!("Hello, world!");
+}

src/build_context.rs

@@ -3,9 +3,9 @@ use auditwheel_rs;
 #[cfg(feature = "sdist")]
 use build_source_distribution;
 use compile;
-use failure::{Error, ResultExt};
+use failure::{Context, Error, ResultExt};
 use module_writer::WheelWriter;
-use module_writer::{write_bindings_module, write_cffi_module};
+use module_writer::{write_bin, write_bindings_module, write_cffi_module};
 use std::collections::HashMap;
 use std::fs;
 use std::path::PathBuf;
@@ -17,17 +17,22 @@ use Target;
 /// pyo3/rust-cpython
 #[derive(Clone, Debug, PartialEq, Eq)]
 pub enum BridgeModel {
+    /// A native module with c bindings, i.e. `#[no_mangle] extern "C" <some item>`
     Cffi,
+    /// A rust binary to be shipped a python package
+    Bin,
+    /// A native module with pyo3 or rust-cpython bindings
     Bindings {
         interpreter: Vec<PythonInterpreter>,
         bindings_crate: String,
     },
 }
 
 impl BridgeModel {
-    fn to_option(&self) -> Option<String> {
+    // TODO
+    pub fn to_option(&self) -> Option<String> {
         match self {
-            BridgeModel::Cffi => None,
+            BridgeModel::Cffi | BridgeModel::Bin => None,
             BridgeModel::Bindings { bindings_crate, .. } => Some(bindings_crate.clone()),
         }
     }
@@ -71,6 +76,7 @@ impl BuildContext {
 
         let wheels = match &self.bridge {
             BridgeModel::Cffi => vec![(self.build_cffi_wheel()?, None)],
+            BridgeModel::Bin => vec![(self.build_bin_wheel()?, None)],
             BridgeModel::Bindings { interpreter, .. } => self.build_binding_wheels(&interpreter)?,
         };
 
@@ -90,13 +96,7 @@ impl BuildContext {
     ) -> Result<Vec<(PathBuf, Option<PythonInterpreter>)>, Error> {
         let mut wheels = Vec::new();
         for python_version in interpreter {
-            let artifact = compile(&self, Some(&python_version), self.bridge.to_option())
-                .context("Failed to build a native library through cargo")?;
-
-            if !self.skip_auditwheel && python_version.target.is_linux() {
-                #[cfg(feature = "auditwheel")]
-                auditwheel_rs(&artifact).context("Failed to ensure manylinux compliance")?;
-            }
+            let artifact = self.compile_cdylib(Some(&python_version))?;
 
             let tag = python_version.get_tag();
 
@@ -138,21 +138,48 @@ impl BuildContext {
         Ok(wheels)
     }
 
-    /// Builds a wheel with cffi bindings
-    pub fn build_cffi_wheel(&self) -> Result<PathBuf, Error> {
-        let artifact = compile(&self, None, self.bridge.to_option())
+    /// Runs cargo build, extracts the cdylib from the output, runs auditwheel and returns the
+    /// artifact
+    pub fn compile_cdylib(
+        &self,
+        python_interpreter: Option<&PythonInterpreter>,
+    ) -> Result<PathBuf, Error> {
+        let artifacts = compile(&self, python_interpreter, self.bridge.to_option())
             .context("Failed to build a native library through cargo")?;
 
-        if !self.skip_auditwheel && self.target.is_linux() {
+        let artifact = artifacts.get("cdylib").cloned().ok_or_else(|| {
+            Context::new(
+                "Cargo didn't build a cdylib. Did you miss crate-type = [\"cdylib\"] \
+                 in the lib section of your Cargo.toml?",
+            )
+        })?;
+
+        let target = python_interpreter
+            .map(|x| &x.target)
+            .unwrap_or(&self.target);
+
+        if !self.skip_auditwheel && target.is_linux() {
             #[cfg(feature = "auditwheel")]
             auditwheel_rs(&artifact).context("Failed to ensure manylinux compliance")?;
         }
 
+        Ok(artifact)
+    }
+
+    fn get_unversal_tags(&self) -> (String, &[&'static str]) {
         let tag = format!(
             "py2.py3-none-{platform}",
             platform = self.target.get_platform_tag()
         );
         let tags = self.target.get_cffi_tags();
+        (tag, tags)
+    }
+
+    /// Builds a wheel with cffi bindings
+    pub fn build_cffi_wheel(&self) -> Result<PathBuf, Error> {
+        let artifact = self.compile_cdylib(None)?;
+
+        let (tag, tags) = self.get_unversal_tags();
 
         let mut builder = WheelWriter::new(&tag, &self.out, &self.metadata21, &self.scripts, tags)?;
 
@@ -164,4 +191,41 @@ impl BuildContext {
 
         Ok(wheel_path)
     }
+
+    /// Builds a wheel that contains a rust binary and an entrypoint for that binary
+    pub fn build_bin_wheel(&self) -> Result<PathBuf, Error> {
+        let artifacts = compile(&self, None, self.bridge.to_option())
+            .context("Failed to build a native library through cargo")?;
+
+        let artifact = artifacts
+            .get("bin")
+            .cloned()
+            .ok_or_else(|| Context::new("Cargo didn't build a binary."))?;
+
+        if !self.skip_auditwheel && self.target.is_linux() {
+            #[cfg(feature = "auditwheel")]
+            auditwheel_rs(&artifact).context("Failed to ensure manylinux compliance")?;
+        }
+
+        let (tag, tags) = self.get_unversal_tags();
+
+        if !self.scripts.is_empty() {
+            bail!("Defining entrypoints and working with a binary doesn't mix well");
+        }
+
+        let mut builder = WheelWriter::new(&tag, &self.out, &self.metadata21, &self.scripts, tags)?;
+
+        // I wouldn't know of any case where this would be the wrong (and neither do
+        // I know a better alternative)
+        let bin_name = artifact
+            .file_name()
+            .expect("Couldn't get the filename from the binary produced by cargo");
+        write_bin(&mut builder, &artifact, &self.metadata21, bin_name)?;
+
+        let wheel_path = builder.finish()?;
+
+        println!("Built wheel to {}", wheel_path.display());
+
+        Ok(wheel_path)
+    }
 }