Skip to content

Commit

Permalink
Merge branch 'master' into export-saved-model
Browse files Browse the repository at this point in the history
  • Loading branch information
@adamcrume
adamcrume committed Mar 11, 2017
2 parents 91d2a39 + 022f4d0 commit e0f4612
Show file tree
Hide file tree
Showing 12 changed files with 1,175 additions and 644 deletions.
6 changes: 4 additions & 2 deletions .travis.yml
View file
Original file line number Diff line number Diff line change
@@ -1,23 +1,25 @@
language: rust
sudo: false
dist: trusty # still in beta, but required for the prebuilt TF binaries

cache:
cargo: true
directories:
- $HOME/.cache/bazel

rust: nightly
rust: stable

install:
- export CC="gcc-4.9" CXX="g++-4.9"
- source travis-ci/install.sh

script:
- export RUST_BACKTRACE=1
- cargo test -vv -j 2 --features tensorflow_unstable
- cargo run --example regression
- cargo run --features tensorflow_unstable --example expressions
- cargo doc -vv --features tensorflow_unstable
- (cd tensorflow-sys && cargo test -vv -j 1)
- # TODO(#66): Re-enable: (cd tensorflow-sys && cargo test -vv -j 1)
- (cd tensorflow-sys && cargo doc -vv)

addons:
Expand Down
116 changes: 66 additions & 50 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,90 +1,105 @@
#TensorFlow Rust
# <img alt="TensorFlow" src="https://www.tensorflow.org/images/tf_logo_transp.png" width="170"/> Rust Binding
[![Version](https://img.shields.io/crates/v/tensorflow.svg)](https://crates.io/crates/tensorflow)
[![Status](https://travis-ci.org/tensorflow/rust.svg?branch=master)](https://travis-ci.org/tensorflow/rust)

TensorFlow Rust provides idiomatic [Rust](https://www.rust-lang.org) language
bindings for [TensorFlow](http://tensorflow.org).

This project is still under active development and not guaranteed to have a
stable API. This is especially true because the TensorFlow C API used by this
project has not yet stabilized.
**Notice:** This project is still under active development and not guaranteed to have a
stable API. This is especially true because the underlying TensorFlow C API has not yet
been stabilized as well.

## Using
* [Documentation](https://tensorflow.github.io/rust/tensorflow/)
* [TensorFlow website](http://tensorflow.org)
* [TensorFlow GitHub page](https://github.com/tensorflow/tensorflow)

## Getting Started
Since this crate depends on the TensorFlow C API, it needs to be compiled first. This crate will
automatically compile TensorFlow for you, but it is also possible to manually install TensorFlow
and the crate will pick it up accordingly.

### Prerequisites
The following dependencies are needed to compile and build this crate (assuming TensorFlow itself
should also be compiled transparently):

- git
- [bazel](https://bazel.build/)
- Python Dependencies `numpy`, `dev`, `pip` and `wheel`
- Optionally, CUDA packages to support GPU-based processing

If you only wish to use TensorFlow within your own project, simply include
`tensorflow = "*"` in your `Cargo.toml` file. If the TensorFlow library is
not already installed on your computer in a place it can be found, you will
need the requirements listed in "Automatically Building Tensorflow" below,
and Cargo will build the library for you.
The TensorFlow website provides detailed instructions on how to obtain and install said dependencies,
so if you are unsure please [check out the docs](https://www.tensorflow.org/install/install_sources)
for further details.

## Building
### Usage
Add this to your `Cargo.toml`:

If you only intend to use TensorFlow from within Rust, then you don't need to
build TensorFlow manually and can follow the automatic steps. If you do need to
use TensorFlow outside of Rust, the manual steps will provide you with a
TensorFlow header file and shared library that can be used by other languages.
```toml
[dependencies]
tensorflow = "0.3"
```

### Automatically building TensorFlow
and this to your crate root:

```rust
extern crate tensorflow;
```

Install [SWIG](http://www.swig.org) and [NumPy](http://www.numpy.org). The
version from your distro's package manager should be fine for these two. Also
install [Bazel](http://bazel.io/docs/install.html), which you may need to do
from source.
Then run `cargo build -j 1`. Since TensorFlow is built during this process, and
the TensorFlow build is very memory intensive, we recommend using the `-j 1`
flag which tells cargo to use only one task, which in turn tells TensorFlow to
build with only on task. Of course, if you have a lot of RAM, you can use a
higher value.

To include the especially unstable API (which is currently the `expr` module),
use `--features tensorflow_unstable`.

### Manually building TensorFlow
For now, please see the [Examples](https://github.com/tensorflow/rust/tree/master/examples) for more
details on how to use this binding.

Install [TensorFlow from source](https://www.tensorflow.org/versions/master/get_started/os_setup.html#source).
## Manual TensorFlow Compilation
If you don't want to build TensorFlow after every `cargo clean` or you want to work against
unreleased/unsupported TensorFlow versions, manual compilation is the way to go.

See [TensorFlow from source](https://www.tensorflow.org/install/install_sources) first.
The Python/pip steps are not necessary, but building `tensorflow:libtensorflow.so` is.

In short:

1. Install [SWIG](http://www.swig.org) and [NumPy](http://www.numpy.org). The
version from your distro's package manager should be fine for these two.
1. [Install Bazel](http://bazel.io/docs/install.html), which you may need to do
2. [Install Bazel](http://bazel.io/docs/install.html), which you may need to do
from source.
1. `git clone --recurse-submodules https://github.com/tensorflow/tensorflow`
1. `cd tensorflow`
1. `./configure`
1. `bazel build -c opt --jobs=1 tensorflow:libtensorflow.so`
3. `git clone https://github.com/tensorflow/tensorflow`
4. `cd tensorflow`
5. `./configure`
6. `bazel build --compilation_mode=opt --copt=-march=native --jobs=1 tensorflow:libtensorflow.so`

Using --jobs=1 is recommended unless you have a lot of RAM, because
Using `--jobs=1` is recommended unless you have a lot of RAM, because
TensorFlow's build is very memory intensive.

Copy $TENSORFLOW_SRC/bazel-bin/tensorflow/libtensorflow.so to /usr/local/lib.
If this is not possible, add $TENSORFLOW_SRC/bazel-bin/tensorflow to
LD_LIBRARY_PATH.
Copy `$TENSORFLOW_SRC/bazel-bin/tensorflow/libtensorflow.so` to `/usr/local/lib`.
If this is not possible, add `$TENSORFLOW_SRC/bazel-bin/tensorflow` to
`LD_LIBRARY_PATH`.

You may need to run `ldconfig` to reset `ld`'s cache after copying libtensorflow.so.
You may need to run `ldconfig` to reset `ld`'s cache after copying `libtensorflow.so`.

Now run `cargo build` as usual.
To include the especially unstable API (which is currently the `expr` module),
use `--features tensorflow_unstable`.

## RFCs
RFCs are [issues tagged with RFC](https://github.com/tensorflow/rust/labels/rfc).
Check them out and comment. Discussions are welcome. After all, thats what a Request For Comment is for!
**OSX Note**: If you are running on OSX, there is a
[Homebrew PR](https://github.com/Homebrew/homebrew-core/pull/10273) in process which, once merged,
will make it easy to install `libtensorflow` wihout hassle. In the meantime, you can take a look at
[snipsco/tensorflow-build](https://github.com/snipsco/tensorflow-build) which provides a homebrew
tap that does essentially the same.

## FAQs
## FAQ's

#### Why does the compiler say that parts of the API don't exist?
### Why does the compiler say that parts of the API don't exist?
The especially unstable parts of the API (which is currently the `expr` modul) are
feature-gated behind the feature `tensorflow_unstable` to prevent accidental
use. See http://doc.crates.io/manifest.html#the-features-section.
(We would prefer using an `#[unstable]` attribute, but that
[doesn't exist](https://github.com/rust-lang/rfcs/issues/1491) yet.)

## Other

This project is not directly affiliated with the TensorFlow project, although we
do intend to communicate and cooperate with them.

## Contributing
Developers and users are welcome to join
[#tensorflow-rust](http://chat.mibbit.com/?server=irc.mozilla.org&channel=%23tensorflow-rust)
on irc.mozilla.org.
Expand All @@ -93,8 +108,9 @@ See [CONTRIBUTING.md](CONTRIBUTING.md) for information on how to contribute code

This is not an official Google product.

##For more information
RFCs are [issues tagged with RFC](https://github.com/tensorflow/rust/labels/rfc).
Check them out and comment. Discussions are welcome. After all, thats what a Request For
Comment is for!

* [API docs](https://tensorflow.github.io/rust)
* [TensorFlow website](http://tensorflow.org)
* [TensorFlow GitHub page](https://github.com/tensorflow/tensorflow)
## License
This project is licensed under the terms of the [Apache 2.0 license](https://github.com/tensorflow/rust/blob/master/LICENSE).
9 changes: 5 additions & 4 deletions src/buffer.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ use std::ops::Range;
use std::ops::RangeFrom;
use std::ops::RangeFull;
use std::ops::RangeTo;
use std::os::raw::c_void as std_c_void;
use std::ptr;
use std::slice;
use super::BufferTrait;
Expand Down Expand Up @@ -65,7 +66,7 @@ impl<T: TensorType> Buffer<T> {
let msg = CStr::from_ptr(c_msg);
panic!("Failed to allocate: {}", msg.to_str().unwrap());
}
(*inner).data = ptr;
(*inner).data = ptr as *mut std_c_void;
(*inner).length = len;
(*inner).data_deallocator = Some(deallocator);
Buffer {
Expand All @@ -81,7 +82,7 @@ impl<T: TensorType> Buffer<T> {
/// The underlying data is *not* freed when the buffer is destroyed.
pub unsafe fn from_ptr(ptr: *mut T, len: usize) -> Self {
let inner = tf::TF_NewBuffer();
(*inner).data = ptr as *const c_void;
(*inner).data = ptr as *const std_c_void;
(*inner).length = len;
Buffer {
inner: inner,
Expand Down Expand Up @@ -155,8 +156,8 @@ impl<T: TensorType> BufferTrait for Buffer<T> {
}
}

unsafe extern "C" fn deallocator(data: *mut c_void, _length: size_t) {
libc::free(data);
unsafe extern "C" fn deallocator(data: *mut std_c_void, _length: size_t) {
libc::free(data as *mut c_void);
}

impl<T: TensorType> Drop for Buffer<T> {
Expand Down
33 changes: 17 additions & 16 deletions src/graph.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ use std;
use std::ffi::CStr;
use std::ffi::CString;
use std::ffi::NulError;
use std::os::raw::c_void as std_c_void;
use std::ptr;
use std::str::Utf8Error;
use std::sync::Arc;
Expand Down Expand Up @@ -302,7 +303,7 @@ impl Operation {
pub fn output_type(&self, index: usize) -> DataType {
unsafe {
DataType::from_c(tf::TF_OperationOutputType(tf::TF_Output {
operation: self.inner,
oper: self.inner,
index: index as c_int,
}))
}
Expand Down Expand Up @@ -332,7 +333,7 @@ impl Operation {
pub fn input_type(&self, index: usize) -> DataType {
unsafe {
DataType::from_c(tf::TF_OperationInputType(tf::TF_Input {
operation: self.inner,
oper: self.inner,
index: index as c_int,
}))
}
Expand All @@ -359,11 +360,11 @@ impl Operation {
pub fn input(&self, index: usize) -> (Operation, usize) {
unsafe {
let port = tf::TF_OperationInput(tf::TF_Input {
operation: self.inner,
oper: self.inner,
index: index as c_int,
});
(Operation {
inner: port.operation,
inner: port.oper,
gimpl: self.gimpl.clone(),
},
port.index as usize)
Expand All @@ -374,7 +375,7 @@ impl Operation {
pub fn output_num_consumers(&self, index: usize) -> usize {
unsafe {
tf::TF_OperationOutputNumConsumers(tf::TF_Output {
operation: self.inner,
oper: self.inner,
index: index as c_int,
}) as usize
}
Expand All @@ -387,12 +388,12 @@ impl Operation {
pub fn output_consumers(&self, index: usize) -> Vec<(Operation, usize)> {
unsafe {
let num_consumers = tf::TF_OperationOutputNumConsumers(tf::TF_Output {
operation: self.inner,
oper: self.inner,
index: index as c_int,
});
let mut vec = <Vec<tf::TF_Input>>::with_capacity(num_consumers as usize);
let len = tf::TF_OperationOutputConsumers(tf::TF_Output {
operation: self.inner,
oper: self.inner,
index: index as c_int,
},
vec.as_mut_ptr(),
Expand All @@ -401,7 +402,7 @@ impl Operation {
vec.into_iter()
.map(|port| {
(Operation {
inner: port.operation,
inner: port.oper,
gimpl: self.gimpl.clone(),
},
port.index as usize)
Expand Down Expand Up @@ -483,7 +484,7 @@ pub struct Input<'a> {
impl<'a> Input<'a> {
fn to_c(&self) -> tf::TF_Input {
tf::TF_Input {
operation: self.operation.inner,
oper: self.operation.inner,
index: self.index,
}
}
Expand All @@ -505,7 +506,7 @@ pub struct Output<'a> {
impl<'a> Output<'a> {
fn to_c(&self) -> tf::TF_Output {
tf::TF_Output {
operation: self.operation.inner,
oper: self.operation.inner,
index: self.index,
}
}
Expand Down Expand Up @@ -606,7 +607,7 @@ impl<'a> OperationDescription<'a> {
unsafe {
tf::TF_SetAttrString(self.inner,
c_attr_name.as_ptr(),
c_value.as_ptr() as *const c_void,
c_value.as_ptr() as *const std_c_void,
c_value.len() as size_t);
}
Ok(())
Expand All @@ -625,7 +626,7 @@ impl<'a> OperationDescription<'a> {
unsafe {
tf::TF_SetAttrStringList(self.inner,
c_attr_name.as_ptr(),
ptrs.as_ptr(),
ptrs.as_ptr() as *const *const std_c_void,
lens.as_ptr(),
ptrs.len() as c_int);
}
Expand Down Expand Up @@ -821,7 +822,7 @@ impl<'a> OperationDescription<'a> {
unsafe {
tf::TF_SetAttrTensorShapeProto(self.inner,
c_attr_name.as_ptr(),
value.as_ptr() as *const c_void,
value.as_ptr() as *const std_c_void,
value.len() as size_t,
status.inner());
}
Expand All @@ -843,7 +844,7 @@ impl<'a> OperationDescription<'a> {
unsafe {
tf::TF_SetAttrTensorShapeProtoList(self.inner,
c_attr_name.as_ptr(),
ptrs.as_ptr(),
ptrs.as_ptr() as *const *const std_c_void,
lens.as_ptr(),
ptrs.len() as c_int,
status.inner());
Expand Down Expand Up @@ -878,7 +879,7 @@ impl<'a> OperationDescription<'a> {
let ptrs: Vec<*mut tf::TF_Tensor> = value.into_iter().map(|x| x.into_ptr()).collect();
tf::TF_SetAttrTensorList(self.inner,
c_attr_name.as_ptr(),
ptrs.as_ptr(),
ptrs.as_ptr() as *const *const tf::TF_Tensor,
ptrs.len() as c_int,
status.inner());
}
Expand All @@ -893,7 +894,7 @@ impl<'a> OperationDescription<'a> {
unsafe {
tf::TF_SetAttrValueProto(self.inner,
c_attr_name.as_ptr(),
value.as_ptr() as *const c_void,
value.as_ptr() as *const std_c_void,
// Allow trivial_numeric_casts because usize is not
// necessarily size_t.
value.len() as size_t,
Expand Down
5 changes: 3 additions & 2 deletions src/lib.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@ use std::ops::Deref;
use std::ops::DerefMut;
use std::ops::Drop;
use std::ops::Index;
use std::os::raw::c_void as std_c_void;
use std::str::Utf8Error;

////////////////////////
Expand Down Expand Up @@ -844,9 +845,9 @@ pub struct Tensor<T: TensorType> {
owned: bool,
}

unsafe extern "C" fn noop_deallocator(_: *mut c_void, _: size_t, _: *mut c_void) -> () {}
unsafe extern "C" fn noop_deallocator(_: *mut std_c_void, _: size_t, _: *mut std_c_void) -> () {}

unsafe extern "C" fn deallocator(_: *mut c_void, _: size_t, buffer: *mut c_void) -> () {
unsafe extern "C" fn deallocator(_: *mut std_c_void, _: size_t, buffer: *mut std_c_void) -> () {
tf::TF_DeleteBuffer(buffer as *mut tf::TF_Buffer);
}

Expand Down
Loading

0 comments on commit e0f4612

Please sign in to comment.