Skip to content
Branch: master
Find file History

Latest commit

crazydonkey200 and Copybara-Service Add instructions for baseline experiments and make error message more…
… informative.

PiperOrigin-RevId: 305120286
Latest commit 3627f05 Apr 7, 2020

Files

Permalink
Type Name Latest commit message Commit time
..
Failed to load latest commit information.
BUILD Fix issues with copy constructor in the unit tests. Mar 7, 2020
README.md Add instructions for baseline experiments and make error message more… Apr 6, 2020
WORKSPACE Update setup.sh, g++ requirements, and eigen_archive to solve depende… Mar 31, 2020
algorithm.cc Ensure random seed is not zero, make printout like algorithms in paper. Mar 5, 2020
algorithm.h Simplify dependencies. Mar 5, 2020
algorithm.proto Simplify dependencies and add bazel WORKSPACE and BUILD files. Mar 5, 2020
algorithm_test.cc Adjust nomenclature to match paper (continued). Mar 5, 2020
algorithm_test_util.cc Internal change Apr 3, 2020
algorithm_test_util.h Fix C++ compiler mismatch issues. Mar 5, 2020
best_algo.gif Update README. Update the best algorithm gif. Mar 9, 2020
compute_cost.cc Simplify dependencies and add bazel WORKSPACE and BUILD files. Mar 5, 2020
compute_cost.h Simplify dependencies. Mar 5, 2020
definitions.h Ensure random seed is not zero, make printout like algorithms in paper. Mar 5, 2020
definitions_test.cc Fix issues with copy constructor in the unit tests. Mar 7, 2020
eigen.BUILD Simplify dependencies and add bazel WORKSPACE and BUILD files. Mar 5, 2020
evaluator.cc Limit experiments by number of train steps as opposed to number of in… Mar 6, 2020
evaluator.h Limit experiments by number of train steps as opposed to number of in… Mar 6, 2020
evaluator_test.cc Fix issues with copy constructor in the unit tests. Mar 7, 2020
executor.h Limit experiments by number of train steps as opposed to number of in… Mar 6, 2020
executor_test.cc Fix issues with copy constructor in the unit tests. Mar 7, 2020
experiment.proto Limit experiments by number of train steps as opposed to number of in… Mar 6, 2020
experiment_progress.png Add figures. Mar 9, 2020
experiment_util.cc Simplify dependencies. Mar 5, 2020
experiment_util.h Simplify dependencies. Mar 5, 2020
fec_cache.cc Simplify code, make nomenclature more like paper. Mar 5, 2020
fec_cache.h Simplify dependencies. Mar 5, 2020
fec_cache.proto Simplify code, make nomenclature more like paper. Mar 5, 2020
fec_cache_test.cc Simplify dependencies and add bazel WORKSPACE and BUILD files. Mar 5, 2020
fec_hashing.cc Simplify hash mix. Mar 4, 2020
fec_hashing.h Simplify dependencies. Mar 5, 2020
fec_hashing_test.cc Simplify namespace. Mar 3, 2020
generate_datasets.py Add support for projected binary classification datasets and a setup … Mar 26, 2020
generator.cc Simplify dependencies and add bazel WORKSPACE and BUILD files. Mar 5, 2020
generator.h Fix issues with copy constructor in the unit tests. Mar 7, 2020
generator.proto Simplify flags and enable FEC in demo. Mar 4, 2020
generator_test.cc Fix issues with copy constructor in the unit tests. Mar 7, 2020
generator_test_util.cc Fix issues with copy constructor in the unit tests. Mar 7, 2020
generator_test_util.h Fix issues with copy constructor in the unit tests. Mar 7, 2020
initial_and_evolved_code.png Add figures. Mar 9, 2020
instruction.cc Ensure random seed is not zero, make printout like algorithms in paper. Mar 5, 2020
instruction.h Simplify dependencies. Mar 5, 2020
instruction.proto Reorganize enums. Mar 4, 2020
instruction_test.cc Fix issues with copy constructor in the unit tests. Mar 7, 2020
memory.cc Clean up dependencies. Mar 4, 2020
memory.h Simplify dependencies. Mar 5, 2020
memory_test.cc Simplify namespace. Mar 3, 2020
mutator.cc Simplify code, make nomenclature more like paper. Mar 5, 2020
mutator.h Fix issues with copy constructor in the unit tests. Mar 7, 2020
mutator.proto Simplify code, make nomenclature more like paper. Mar 5, 2020
mutator_test.cc Fix issues with copy constructor in the unit tests. Mar 7, 2020
progress.gif Update progress.gif to make curve more distinguishable. Mar 9, 2020
random_generator.cc Ensure random seed is not zero, make printout like algorithms in paper. Mar 5, 2020
random_generator.h Fix issues with copy constructor in the unit tests. Mar 7, 2020
random_generator_test.cc Simplify namespace. Mar 3, 2020
randomizer.cc Simplify namespace. Mar 3, 2020
randomizer.h Simplify dependencies. Mar 5, 2020
regularized_evolution.cc Limit experiments by number of train steps as opposed to number of in… Mar 6, 2020
regularized_evolution.h Limit experiments by number of train steps as opposed to number of in… Mar 6, 2020
regularized_evolution_test.cc Fix issues with copy constructor in the unit tests. Mar 7, 2020
run_baseline.sh Less printouts for baseline. Apr 2, 2020
run_demo.sh Fix issues with copy constructor in the unit tests. Mar 7, 2020
run_integration_test_linear.sh Fix issues with copy constructor in the unit tests. Mar 7, 2020
run_integration_test_nonlinear.sh Fix issues with copy constructor in the unit tests. Mar 7, 2020
run_integration_test_projected_binary_datasets.sh Add instructions for baseline experiments and make error message more… Apr 6, 2020
run_integration_tests.sh Fix C++ compiler mismatch issues. Mar 5, 2020
run_search_experiment.cc Limit experiments by number of train steps as opposed to number of in… Mar 6, 2020
setup.sh Add instructions for baseline experiments and make error message more… Apr 6, 2020
task.h Fix C++ compiler mismatch issues. Mar 5, 2020
task.proto Add support for projected binary classification datasets and a setup … Mar 26, 2020
task_pb2.py Add task_pb2.py to let the user avoid the trouble of compiling protos… Mar 26, 2020
task_test.cc Adjust nomenclature to match paper (continued). Mar 5, 2020
task_util.cc Fix C++ compiler mismatch issues. Mar 5, 2020
task_util.h Add instructions for baseline experiments and make error message more… Apr 6, 2020
task_util_test.cc Fix issues with copy constructor in the unit tests. Mar 7, 2020
test_util.h Simplify dependencies. Mar 5, 2020
test_util_test.cc Fix C++ compiler mismatch issues. Mar 5, 2020
train_budget.cc Simplify flags and enable FEC in demo. Mar 4, 2020
train_budget.h Simplify dependencies. Mar 5, 2020
train_budget.proto Simplify dependencies and add bazel WORKSPACE and BUILD files. Mar 5, 2020
train_budget_test.cc Simplify namespace. Mar 3, 2020
util.cc Simplify namespace. Mar 3, 2020
util.h Simplify dependencies. Mar 5, 2020
util_test.cc Simplify namespace. Mar 3, 2020

README.md

AutoML-Zero

Open source code for the paper: "AutoML-Zero: Evolving Machine Learning Algorithms From Scratch"

Introduction Quick Demo Reproducing Search Baselines Citation

What is AutoML-Zero?

AutoML-Zero aims to automatically discover computer programs that can solve machine learning tasks, starting from empty or random programs and using only basic math operations. The goal is to simultaneously search for all aspects of an ML algorithm—including the model structure and the learning strategy—while employing minimal human bias.

GIF for the experiment progress

Despite AutoML-Zero's challenging search space, evolutionary search shows promising results by discovering linear regression with gradient descent, 2-layer neural networks with backpropagation, and even algorithms that surpass hand designed baselines of comparable complexity. The figure above shows an example sequence of discoveries from one of our experiments, evolving algorithms to solve binary classification tasks. Notably, the evolved algorithms can be interpreted. Below is an analysis of the best evolved algorithm: the search process "invented" techniques like bilinear interactions, weight averaging, normalized gradient, and data augmentation (by adding noise to the inputs).

GIF for the interpretation of the best evolved algorithm

More examples, analysis, and details can be found in the paper.

 

5-Minute Demo: Discovering Linear Regression From Scratch

As a miniature "AutoML-Zero" experiment, let's try to automatically discover programs to solve linear regression tasks.

To get started, first install bazel following instructions here (bazel>=2.2.0 and g++>=9 are required), then run the demo with:

git clone https://github.com/google-research/google-research.git
cd google-research/automl_zero
./run_demo.sh

This script runs evolutionary search on 10 linear tasks (Tsearch in the paper). After each experiment, it evaluates the best algorithm discovered on 100 new linear tasks (Tselect in the paper). Once an algorithm attains a fitness (1 - RMS error) greater than 0.9999, it is selected for a final evaluation on 100 unseen tasks. To conclude, the demo prints the results of the final evaluation and shows the code for the automatically discovered algorithm.

To make this demo quick, we use a much smaller search space than in the paper: only the math operations necessary to implement linear regression are allowed and the programs are constrained to a short, fixed length. Even with these limitations, the search space is quite sparse, as random search experiments show that only ~1 in 108 algorithms in the space can solve the tasks with the required accuracy. Nevertheless, this demo typically discovers programs similar to linear regression by gradient descent in under 5 minutes using 1 CPU (Note that the runtime may vary due to random seeds and hardware). We have seen similar and more interesting discoveries in the unconstrained search space (see more details in the paper).

You can compare the automatically discovered algorithm with the solution from a human ML researcher (one of the authors):

def Setup():
  s2 = 0.001  # Init learning rate.

def Predict():  # v0 = features
  s1 = dot(v0, v1)  # Apply weights

def Learn():  # v0 = features; s0 = label
  s3 = s0 - s1  # Compute error.
  s4 = s3 * s1  # Apply learning rate.
  v2 = v0 * s4  # Compute gradient.
  v1 = v1 + v2  # Update weights.

In this human designed program, the Setup function establishes a learning rate, the Predict function applies a set of weights to the inputs, and the Learn function corrects the weights in the opposite direction to the gradient; in other words, a linear regressor trained with gradient descent. The evolved programs may look different even if they have the same functionality due to redundant instructions and different ordering, which can make them challenging to interpret. See more details about how we address these problems in the paper.

 

Reproducing Search Baselines

First install bazel, following the instructions here (bazel>=2.2.0 and g++>=9 are required), then follow the instructions below to reproduce the results in Supplementary Section 9 ("Baselines") with the "Basic" method on 1 process (1 CPU).

First, generate the projected binary CIFAR10 datasets by running

python generate_datasets.py --data_dir=binary_cifar10_data

It takes ~1 hrs to download and preprocess all the data.

Then, start the baseline experiment by running

./run_baseline.sh

It takes 12-18 hrs to finish, depending on the hardware. You can vary the random seed in run_baseline.sh to produce a different result for each run.

If you want to use more than 1 process, you will need to create your own implementation to parallelize the computation based on your particular distributed-computing platform. A platform-agnostic description of what we did is given in our paper.

Note we left out of this directory upgrades for the "Full" method that are pre-existing (hurdles) but included those introduced in this paper (e.g. FEC for ML algorithms).

Citation

If you use the code in your research, please cite:

@article{real2020automl,
  title={AutoML-Zero: Evolving Machine Learning Algorithms From Scratch},
  author={Real, Esteban and Liang, Chen and So, David R and Le, Quoc V},
  journal={arXiv preprint arXiv:2003.03384},
  year={2020}
}

 

Search keywords: machine learning, neural networks, evolution, evolutionary algorithms, regularized evolution, program synthesis, architecture search, NAS, neural architecture search, neuro-architecture search, AutoML, AutoML-Zero, algorithm search, meta-learning, genetic algorithms, genetic programming, neuroevolution, neuro-evolution.

You can’t perform that action at this time.