Skip to content

Commit

Permalink
Merge changes from github.
Browse files Browse the repository at this point in the history 
PiperOrigin-RevId: 188817194
  • Loading branch information
@caisq @tensorflower-gardener
caisq authored and tensorflower-gardener committed Mar 13, 2018
1 parent 2bda52d commit 7144571
Show file tree
Hide file tree
Showing 103 changed files with 4,051 additions and 802 deletions.
37 changes: 20 additions & 17 deletions README.md
View file
Expand Up @@ -4,9 +4,10 @@

-----------------

| **`Linux CPU`** | **`Linux GPU`** | **`Mac OS CPU`** | **`Windows CPU`** | **`Android`** |
|-----------------|---------------------|------------------|-------------------|---------------|
| [![Build Status](https://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-cpu)](https://ci.tensorflow.org/job/tensorflow-master-cpu) | [![Build Status](https://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-linux-gpu)](https://ci.tensorflow.org/job/tensorflow-master-linux-gpu) | [![Build Status](https://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-mac)](https://ci.tensorflow.org/job/tensorflow-master-mac) | [![Build Status](https://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-win-cmake-py)](https://ci.tensorflow.org/job/tensorflow-master-win-cmake-py) | [![Build Status](https://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-android)](https://ci.tensorflow.org/job/tensorflow-master-android) [ ![Download](https://api.bintray.com/packages/google/tensorflow/tensorflow/images/download.svg) ](https://bintray.com/google/tensorflow/tensorflow/_latestVersion) |

| **`Documentation`** | **`Linux CPU`** | **`Linux GPU`** | **`Mac OS CPU`** | **`Windows CPU`** | **`Android`** |
|-----------------|---------------------|------------------|-------------------|---------------|---------------|
| [![Documentation](https://img.shields.io/badge/api-reference-blue.svg)](https://www.tensorflow.org/api_docs/) | [![Build Status](https://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-cpu)](https://ci.tensorflow.org/job/tensorflow-master-cpu) | [![Build Status](https://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-linux-gpu)](https://ci.tensorflow.org/job/tensorflow-master-linux-gpu) | [![Build Status](https://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-mac)](https://ci.tensorflow.org/job/tensorflow-master-mac) | [![Build Status](https://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-win-cmake-py)](https://ci.tensorflow.org/job/tensorflow-master-win-cmake-py) | [![Build Status](https://ci.tensorflow.org/buildStatus/icon?job=tensorflow-master-android)](https://ci.tensorflow.org/job/tensorflow-master-android) [ ![Download](https://api.bintray.com/packages/google/tensorflow/tensorflow/images/download.svg) ](https://bintray.com/google/tensorflow/tensorflow/_latestVersion)

**TensorFlow** is an open source software library for numerical computation using
data flow graphs. The graph nodes represent mathematical operations, while
Expand All @@ -21,20 +22,6 @@ organization for the purposes of conducting machine learning and deep neural
networks research. The system is general enough to be applicable in a wide
variety of other domains, as well.

**If you want to contribute to TensorFlow, be sure to review the [contribution
guidelines](CONTRIBUTING.md). This project adheres to TensorFlow's
[code of conduct](CODE_OF_CONDUCT.md). By participating, you are expected to
uphold this code.**

**We use [GitHub issues](https://github.com/tensorflow/tensorflow/issues) for
tracking requests and bugs. So please see
[TensorFlow Discuss](https://groups.google.com/a/tensorflow.org/forum/#!forum/discuss) for general questions
and discussion, and please direct specific questions to [Stack Overflow](https://stackoverflow.com/questions/tagged/tensorflow).**

The TensorFlow project strives to abide by generally accepted best practices in open-source software development:

[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/1486/badge)](https://bestpractices.coreinfrastructure.org/projects/1486)

## Installation
*See [Installing TensorFlow](https://www.tensorflow.org/get_started/os_setup.html) for instructions on how to install our release binaries or how to build from source.*

Expand Down Expand Up @@ -75,6 +62,22 @@ $ python
>>> sess.close()
```

## Contribution guidelines

**If you want to contribute to TensorFlow, be sure to review the [contribution
guidelines](CONTRIBUTING.md). This project adheres to TensorFlow's
[code of conduct](CODE_OF_CONDUCT.md). By participating, you are expected to
uphold this code.**

**We use [GitHub issues](https://github.com/tensorflow/tensorflow/issues) for
tracking requests and bugs. So please see
[TensorFlow Discuss](https://groups.google.com/a/tensorflow.org/forum/#!forum/discuss) for general questions
and discussion, and please direct specific questions to [Stack Overflow](https://stackoverflow.com/questions/tagged/tensorflow).**

The TensorFlow project strives to abide by generally accepted best practices in open-source software development:

[![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/1486/badge)](https://bestpractices.coreinfrastructure.org/projects/1486)

## For more information

* [TensorFlow Website](https://www.tensorflow.org)
Expand Down
6 changes: 3 additions & 3 deletions SECURITY.md
View file
Expand Up @@ -233,7 +233,7 @@ v//Fw6ZeY+HmRDFdirjD7wXtIuER4vqCryIqR6Xe9X8oJXz9L/Jhslc=

### Known vulnerabilities

| Type | Versions affected | Reported by | Additional Information |
|------|:-----------------:|---------------------------------------|
| out of bounds read| <=1.4 | TenCent Blade Team | [issue report](https://github.com/tensorflow/tensorflow/issues/14959) |
| Type | Versions affected | Reported by | Additional Information |
|-------------------|:-----------------:|--------------------|-----------------------------|
| out of bounds read| <=1.4 | TenCent Blade Team | [issue report](https://github.com/tensorflow/tensorflow/issues/14959) |

3 changes: 2 additions & 1 deletion configure
View file
Expand Up @@ -8,7 +8,8 @@ if [ -z "$PYTHON_BIN_PATH" ]; then
fi

# Set all env variables
"$PYTHON_BIN_PATH" configure.py
CONFIGURE_DIR=$(dirname "$0")
"$PYTHON_BIN_PATH" "${CONFIGURE_DIR}/configure.py" "$@"

echo "Configuration finished"

76 changes: 38 additions & 38 deletions configure.py
View file
Expand Up @@ -18,6 +18,7 @@
from __future__ import division
from __future__ import print_function

import argparse
import errno
import os
import platform
Expand All @@ -32,10 +33,6 @@
from distutils.spawn import find_executable as which
# pylint: enable=g-import-not-at-top

_TF_BAZELRC = os.path.join(os.path.dirname(os.path.abspath(__file__)),
'.tf_configure.bazelrc')
_TF_WORKSPACE = os.path.join(os.path.dirname(os.path.abspath(__file__)),
'WORKSPACE')
_DEFAULT_CUDA_VERSION = '9.0'
_DEFAULT_CUDNN_VERSION = '7'
_DEFAULT_CUDA_COMPUTE_CAPABILITIES = '3.5,5.2'
Expand All @@ -51,6 +48,11 @@

_DEFAULT_PROMPT_ASK_ATTEMPTS = 10

_TF_WORKSPACE_ROOT = os.path.abspath(os.path.dirname(__file__))
_TF_BAZELRC_FILENAME = '.tf_configure.bazelrc'
_TF_BAZELRC = os.path.join(_TF_WORKSPACE_ROOT, _TF_BAZELRC_FILENAME)
_TF_WORKSPACE = os.path.join(_TF_WORKSPACE_ROOT, 'WORKSPACE')


class UserInputError(Exception):
pass
Expand Down Expand Up @@ -119,22 +121,6 @@ def sed_in_place(filename, old, new):
f.write(newdata)


def remove_line_with(filename, token):
"""Remove lines that contain token from file.
Args:
filename: string for filename.
token: string token to check if to remove a line from file or not.
"""
with open(filename, 'r') as f:
filedata = f.read()

with open(filename, 'w') as f:
for line in filedata.strip().split('\n'):
if token not in line:
f.write(line + '\n')


def write_to_bazelrc(line):
with open(_TF_BAZELRC, 'a') as f:
f.write(line + '\n')
Expand Down Expand Up @@ -245,33 +231,39 @@ def setup_python(environ_cp):
environ_cp['PYTHON_BIN_PATH'] = python_bin_path

# Write tools/python_bin_path.sh
with open('tools/python_bin_path.sh', 'w') as f:
with open(os.path.join(
_TF_WORKSPACE_ROOT, 'tools', 'python_bin_path.sh'), 'w') as f:
f.write('export PYTHON_BIN_PATH="%s"' % python_bin_path)


def reset_tf_configure_bazelrc():
def reset_tf_configure_bazelrc(workspace_path):
"""Reset file that contains customized config settings."""
open(_TF_BAZELRC, 'w').close()

home = os.path.expanduser('~')
if not os.path.exists('.bazelrc'):
if os.path.exists(os.path.join(home, '.bazelrc')):
with open('.bazelrc', 'a') as f:
f.write('import %s/.bazelrc\n' % home.replace('\\', '/'))
bazelrc_path = os.path.join(workspace_path, '.bazelrc')

data = []
if os.path.exists(bazelrc_path):
with open(bazelrc_path, 'r') as f:
data = f.read().splitlines()
with open(bazelrc_path, 'w') as f:
for l in data:
if _TF_BAZELRC_FILENAME in l:
continue
f.write('%s\n' % l)
if is_windows():
tf_bazelrc_path = _TF_BAZELRC.replace("\\", "/")
else:
open('.bazelrc', 'w').close()

remove_line_with('.bazelrc', 'tf_configure')
with open('.bazelrc', 'a') as f:
f.write('import %workspace%/.tf_configure.bazelrc\n')
tf_bazelrc_path = _TF_BAZELRC
f.write('import %s\n' % tf_bazelrc_path)


def cleanup_makefile():
"""Delete any leftover BUILD files from the Makefile build.
These files could interfere with Bazel parsing.
"""
makefile_download_dir = 'tensorflow/contrib/makefile/downloads'
makefile_download_dir = os.path.join(
_TF_WORKSPACE_ROOT, 'tensorflow', 'contrib', 'makefile', 'downloads')
if os.path.isdir(makefile_download_dir):
for root, _, filenames in os.walk(makefile_download_dir):
for f in filenames:
Expand Down Expand Up @@ -456,7 +448,7 @@ def check_bazel_version(min_version):
if which('bazel') is None:
print('Cannot find bazel. Please install bazel.')
sys.exit(0)
curr_version = run_shell(['bazel', '--batch', 'version'])
curr_version = run_shell(['bazel', '--batch', '--bazelrc=/dev/null', 'version'])

for line in curr_version.split('\n'):
if 'Build label: ' in line:
Expand Down Expand Up @@ -502,7 +494,8 @@ def set_cc_opt_flags(environ_cp):
for opt in cc_opt_flags.split():
write_to_bazelrc('build:opt --copt=%s' % opt)
# It should be safe on the same build host.
write_to_bazelrc('build:opt --host_copt=-march=native')
if not is_ppc64le():
write_to_bazelrc('build:opt --host_copt=-march=native')
write_to_bazelrc('build:opt --define with_default_optimizations=true')
# TODO(mikecase): Remove these default defines once we are able to get
# TF Lite targets building without them.
Expand Down Expand Up @@ -1229,7 +1222,7 @@ def set_host_c_compiler(environ_cp):
environ_cp,
var_name='HOST_C_COMPILER',
var_default=default_c_host_compiler,
ask_for_var=('Please specify which C compiler should be used as the host'
ask_for_var=('Please specify which C compiler should be used as the host '
'C compiler.'),
check_success=os.path.exists,
error_msg='Invalid C compiler path. %s cannot be found.',
Expand Down Expand Up @@ -1373,13 +1366,20 @@ def config_info_line(name, help_text):


def main():
parser = argparse.ArgumentParser()
parser.add_argument("--workspace",
type=str,
default=_TF_WORKSPACE_ROOT,
help="The absolute path to your active Bazel workspace.")
args = parser.parse_args()

# Make a copy of os.environ to be clear when functions and getting and setting
# environment variables.
environ_cp = dict(os.environ)

check_bazel_version('0.5.4')

reset_tf_configure_bazelrc()
reset_tf_configure_bazelrc(args.workspace)
cleanup_makefile()
setup_python(environ_cp)

Expand Down
64 changes: 64 additions & 0 deletions tensorflow/cc/gradients/nn_grad.cc
View file
Expand Up @@ -182,6 +182,70 @@ Status MaxPoolGradV2Helper(const Scope& scope, const Operation& op,
}
REGISTER_GRADIENT_OP("MaxPoolV2", MaxPoolGradV2Helper);

Status MaxPool3DGradHelper(const Scope& scope, const Operation& op,
const std::vector<Output>& grad_inputs,
std::vector<Output>* grad_outputs) {
std::vector<int32> ksize;
std::vector<int32> strides;
string padding;
string data_format;
auto attrs = op.output(0).node()->attrs();
TF_RETURN_IF_ERROR(GetNodeAttr(attrs, "ksize", &ksize));
TF_RETURN_IF_ERROR(GetNodeAttr(attrs, "strides", &strides));
TF_RETURN_IF_ERROR(GetNodeAttr(attrs, "padding", &padding));
TF_RETURN_IF_ERROR(GetNodeAttr(attrs, "data_format", &data_format));
MaxPool3DGrad::Attrs grad_attrs;
auto dx = MaxPool3DGrad(scope, op.input(0), op.output(0), grad_inputs[0],
ksize, strides, padding,
grad_attrs.DataFormat(data_format));
grad_outputs->push_back(dx);
return scope.status();
}
REGISTER_GRADIENT_OP("MaxPool3D", MaxPool3DGradHelper);

Status AvgPoolGradHelper(const Scope& scope, const Operation& op,
const std::vector<Output>& grad_inputs,
std::vector<Output>* grad_outputs) {
std::vector<int32> ksize;
std::vector<int32> strides;
string padding;
string data_format;
auto attrs = op.output(0).node()->attrs();
TF_RETURN_IF_ERROR(GetNodeAttr(attrs, "ksize", &ksize));
TF_RETURN_IF_ERROR(GetNodeAttr(attrs, "strides", &strides));
TF_RETURN_IF_ERROR(GetNodeAttr(attrs, "padding", &padding));
TF_RETURN_IF_ERROR(GetNodeAttr(attrs, "data_format", &data_format));
internal::AvgPoolGrad::Attrs grad_attrs;
auto dx =
internal::AvgPoolGrad(scope, Shape(scope, op.input(0)), grad_inputs[0],
ksize, strides, padding,
grad_attrs.DataFormat(data_format));
grad_outputs->push_back(dx);
return scope.status();
}
REGISTER_GRADIENT_OP("AvgPool", AvgPoolGradHelper);

Status AvgPool3DGradHelper(const Scope& scope, const Operation& op,
const std::vector<Output>& grad_inputs,
std::vector<Output>* grad_outputs) {
std::vector<int32> ksize;
std::vector<int32> strides;
string padding;
string data_format;
auto attrs = op.output(0).node()->attrs();
TF_RETURN_IF_ERROR(GetNodeAttr(attrs, "ksize", &ksize));
TF_RETURN_IF_ERROR(GetNodeAttr(attrs, "strides", &strides));
TF_RETURN_IF_ERROR(GetNodeAttr(attrs, "padding", &padding));
TF_RETURN_IF_ERROR(GetNodeAttr(attrs, "data_format", &data_format));
AvgPool3DGrad::Attrs grad_attrs;
auto dx = AvgPool3DGrad(scope, Shape(scope, op.input(0)), grad_inputs[0],
ksize, strides, padding,
grad_attrs.DataFormat(data_format));
grad_outputs->push_back(dx);
return scope.status();
}
REGISTER_GRADIENT_OP("AvgPool3D", AvgPool3DGradHelper);

Status LRNGradHelper(const Scope& scope, const Operation& op,
const std::vector<Output>& grad_inputs,
std::vector<Output>* grad_outputs) {
Expand Down
44 changes: 41 additions & 3 deletions tensorflow/cc/gradients/nn_grad_test.cc
View file
Expand Up @@ -31,8 +31,11 @@ using ops::Elu;
using ops::L2Loss;
using ops::LogSoftmax;
using ops::LRN;
using ops::AvgPool;
using ops::AvgPool3D;
using ops::MaxPool;
using ops::MaxPoolV2;
using ops::MaxPool3D;
using ops::Placeholder;
using ops::Relu;
using ops::Relu6;
Expand Down Expand Up @@ -70,9 +73,9 @@ class NNGradTest : public ::testing::Test {

// Sets tensor with random values, ensuring that the max value is largest by
// a reasonable amount.
// This is an issue for MaxPool and MaxPoolV2, in which perturbations by the
// numeric gradient computation in the gradient checker can change the max
// value if values are too close together.
// This is an issue for MaxPool, MaxPoolV2 and MaxPool3D, in which
// perturbations by the numeric gradient computation in the gradient checker
// can change the max value if values are too close together.
template <typename T>
void SetRandomValuesWithBumpedMax(Tensor* tensor) {
auto tensor_flat = tensor->flat<T>();
Expand Down Expand Up @@ -203,6 +206,41 @@ TEST_F(NNGradTest, MaxPoolGradV2Helper) {
RunTest(x, x_init_value, y, y_shape);
}

TEST_F(NNGradTest, MaxPool3DGradHelper) {
TensorShape x_shape({1, 3, 3, 3, 1});
TensorShape y_shape({1, 1, 1, 1, 1});
auto x = Placeholder(scope_, DT_FLOAT, Placeholder::Shape(x_shape));
// Setup window and strides so that we only do one MaxPool3D.
const std::vector<int> ksize{1, 3, 3, 3, 1};
const std::vector<int> strides{1, 3, 3, 3, 1};
auto y = MaxPool3D(scope_, x, ksize, strides, "VALID");
Tensor x_init_value = Tensor(DT_FLOAT, x_shape);
SetRandomValuesWithBumpedMax<float>(&x_init_value);
RunTest(x, x_init_value, y, y_shape);
}

TEST_F(NNGradTest, AvgPoolGradHelper) {
TensorShape x_shape({1, 2, 2, 1});
TensorShape y_shape({1, 1, 1, 1});
auto x = Placeholder(scope_, DT_FLOAT, Placeholder::Shape(x_shape));
// Setup window and strides so that we only do one AvgPool.
const std::vector<int> ksize{1, 2, 2, 1};
const std::vector<int> strides{1, 2, 2, 1};
auto y = AvgPool(scope_, x, ksize, strides, "SAME");
RunTest(x, x_shape, y, y_shape);
}

TEST_F(NNGradTest, AvgPool3DGradHelper) {
TensorShape x_shape({1, 3, 3, 3, 1});
TensorShape y_shape({1, 1, 1, 1, 1});
auto x = Placeholder(scope_, DT_FLOAT, Placeholder::Shape(x_shape));
// Setup window and strides so that we only do one AvgPool3D.
const std::vector<int> ksize{1, 3, 3, 3, 1};
const std::vector<int> strides{1, 3, 3, 3, 1};
auto y = AvgPool3D(scope_, x, ksize, strides, "SAME");
RunTest(x, x_shape, y, y_shape);
}

TEST_F(NNGradTest, LRN){
TensorShape x_shape({1, 1, 2, 1});
auto x = Placeholder(scope_, DT_FLOAT, Placeholder::Shape(x_shape));
Expand Down
6 changes: 3 additions & 3 deletions tensorflow/cc/profiler/profiler.h
View file
Expand Up @@ -61,18 +61,18 @@ class Profiler {
/// Adds tracing information `run_meta` to profiler. A `run_meta` is
/// generated by a TensorFlow session run call. `step` is the key
/// to the `run_meta`. When calling ProfileXXX methods, caller can specify
/// `step` in `options` to seletively profile the corresponding `run_meta`.
/// `step` in `options` to selectively profile the corresponding `run_meta`.
/// Multiple different `run_meta` can be keyed by the same `step` in order
/// to group them together.
void AddStep(int64 step, const RunMetadata& run_meta);

/// Profiles the model by organizing nodes in graph structure.
/// Each node is an op and the nodes are contected by the op inputs/outputs.
/// Each node is an op and the nodes are connected by the op inputs/outputs.
GraphNodeProto ProfileGraph(const Options& options);

/// Profiles the model by organizing nodes in name scope structure.
/// Each node is an op, and nodes are organized by the ops' name
/// scope, similar to a filesystem tree.
/// scope, similar to a file system tree.
/// E.g. /foo is the root of operation /foo/matmul_1 and foo/conv_2.
GraphNodeProto ProfileNameScope(const Options& options);

Expand Down

0 comments on commit 7144571

Please sign in to comment.