Building-Recommendation-Engine-Using-Factorization-Machines-PilHoKim-29thAugust2018.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Building Recommendation Engine Using Factorization Machines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare data set using MovieLens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This lab is an extended version of AWS's ML blog [Build a movie recommender with factorization machines on Amazon SageMaker](https://aws.amazon.com/blogs/machine-learning/build-a-movie-recommender-with-factorization-machines-on-amazon-sagemaker/). In the origianl lab available at [GitHub](https://github.com/juliensimon/dlnotebooks/blob/master/sagemaker/03-Factorization-Machines-Movielens.ipynb), it used 100k movie rating data that achieved around 71.5% accuracy.\n",
    "\n",
    "In this lab, we will utilize the same public benchmarking data set known as [MovieLens](https://grouplens.org/datasets/movielens/) but much bigger than the previous one. The data we will use is composed of 20 million ratings and 465,000 tag applications applied to 27,000 movies by 138,000 users. Includes tag genome data with 12 million relevance scores across 1,100 tags.\n",
    "\n",
    "The data structure is a little different with 100k samples. So the below code flow would be a bit different with [the original version]((https://github.com/juliensimon/dlnotebooks/blob/master/sagemaker/03-Factorization-Machines-Movielens.ipynb).\n",
    "\n",
    "Let's see if the bigger data set contributes to the predction accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load python modules and define working environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "import sagemaker.amazon.common as smac\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.predictor import json_deserializer\n",
    "\n",
    "import boto3, csv, io, json\n",
    "import numpy as np\n",
    "from scipy.sparse import lil_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "bucket = 'pilho-sagemaker-ai-workshop'\n",
    "prefix = 'sagemaker/fm-movielens'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Download data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2018-08-29 04:55:47--  http://files.grouplens.org/datasets/movielens/ml-20m.zip\n",
      "Resolving files.grouplens.org (files.grouplens.org)... 128.101.34.235\n",
      "Connecting to files.grouplens.org (files.grouplens.org)|128.101.34.235|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 198702078 (189M) [application/zip]\n",
      "Saving to: ‘/tmp/movielens/ml-20m.zip’\n",
      "\n",
      "ml-20m.zip          100%[===================>] 189.50M  56.8MB/s    in 3.5s    \n",
      "\n",
      "2018-08-29 04:55:51 (54.4 MB/s) - ‘/tmp/movielens/ml-20m.zip’ saved [198702078/198702078]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget -P /tmp/movielens/ http://files.grouplens.org/datasets/movielens/ml-20m.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  /tmp/movielens/ml-20m.zip\n",
      "   creating: /tmp/movielens/data/ml-20m/\n",
      "  inflating: /tmp/movielens/data/ml-20m/genome-scores.csv  \n",
      "  inflating: /tmp/movielens/data/ml-20m/genome-tags.csv  \n",
      "  inflating: /tmp/movielens/data/ml-20m/links.csv  \n",
      "  inflating: /tmp/movielens/data/ml-20m/movies.csv  \n",
      "  inflating: /tmp/movielens/data/ml-20m/ratings.csv  \n",
      "  inflating: /tmp/movielens/data/ml-20m/README.txt  \n",
      "  inflating: /tmp/movielens/data/ml-20m/tags.csv  \n"
     ]
    }
   ],
   "source": [
    "!unzip /tmp/movielens/ml-20m.zip -d/tmp/movielens/data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "movieId,tagId,relevance\r",
      "\r\n",
      "1,1,0.025000000000000022\r",
      "\r\n",
      "1,2,0.025000000000000022\r",
      "\r\n",
      "1,3,0.057750000000000024\r",
      "\r\n",
      "1,4,0.09675\r",
      "\r\n",
      "1,5,0.14675\r",
      "\r\n",
      "1,6,0.21700000000000003\r",
      "\r\n",
      "1,7,0.067\r",
      "\r\n",
      "1,8,0.26275000000000004\r",
      "\r\n",
      "1,9,0.262\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -10 /tmp/movielens/data/ml-20m/genome-scores.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "userId,movieId,tag,timestamp\r",
      "\r\n",
      "18,4141,Mark Waters,1240597180\r",
      "\r\n",
      "65,208,dark hero,1368150078\r",
      "\r\n",
      "65,353,dark hero,1368150079\r",
      "\r\n",
      "65,521,noir thriller,1368149983\r",
      "\r\n",
      "65,592,dark hero,1368150078\r",
      "\r\n",
      "65,668,bollywood,1368149876\r",
      "\r\n",
      "65,898,screwball comedy,1368150160\r",
      "\r\n",
      "65,1248,noir thriller,1368149983\r",
      "\r\n",
      "65,1391,mars,1368150055\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -10 /tmp/movielens/data/ml-20m/tags.csv "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "movieId,imdbId,tmdbId\r",
      "\r\n",
      "1,0114709,862\r",
      "\r\n",
      "2,0113497,8844\r",
      "\r\n",
      "3,0113228,15602\r",
      "\r\n",
      "4,0114885,31357\r",
      "\r\n",
      "5,0113041,11862\r",
      "\r\n",
      "6,0113277,949\r",
      "\r\n",
      "7,0114319,11860\r",
      "\r\n",
      "8,0112302,45325\r",
      "\r\n",
      "9,0114576,9091\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -10 /tmp/movielens/data/ml-20m/links.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "userId,movieId,rating,timestamp\r",
      "\r\n",
      "1,2,3.5,1112486027\r",
      "\r\n",
      "1,29,3.5,1112484676\r",
      "\r\n",
      "1,32,3.5,1112484819\r",
      "\r\n",
      "1,47,3.5,1112484727\r",
      "\r\n",
      "1,50,3.5,1112484580\r",
      "\r\n",
      "1,112,3.5,1094785740\r",
      "\r\n",
      "1,151,4.0,1094785734\r",
      "\r\n",
      "1,223,4.0,1112485573\r",
      "\r\n",
      "1,253,4.0,1112484940\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -10 /tmp/movielens/data/ml-20m/ratings.csv "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "movieId,title,genres\r",
      "\r\n",
      "1,Toy Story (1995),Adventure|Animation|Children|Comedy|Fantasy\r",
      "\r\n",
      "2,Jumanji (1995),Adventure|Children|Fantasy\r",
      "\r\n",
      "3,Grumpier Old Men (1995),Comedy|Romance\r",
      "\r\n",
      "4,Waiting to Exhale (1995),Comedy|Drama|Romance\r",
      "\r\n",
      "5,Father of the Bride Part II (1995),Comedy\r",
      "\r\n",
      "6,Heat (1995),Action|Crime|Thriller\r",
      "\r\n",
      "7,Sabrina (1995),Comedy|Romance\r",
      "\r\n",
      "8,Tom and Huck (1995),Adventure|Children\r",
      "\r\n",
      "9,Sudden Death (1995),Action\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -10 /tmp/movielens/data/ml-20m/movies.csv "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the total number of movies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "131241,Ants in the Pants (2000),Comedy|Romance\r",
      "\r\n",
      "131243,Werner - Gekotzt wird später (2003),Animation|Comedy\r",
      "\r\n",
      "131248,Brother Bear 2 (2006),Adventure|Animation|Children|Comedy|Fantasy\r",
      "\r\n",
      "131250,No More School (2000),Comedy\r",
      "\r\n",
      "131252,Forklift Driver Klaus: The First Day on the Job (2001),Comedy|Horror\r",
      "\r\n",
      "131254,Kein Bund für's Leben (2007),Comedy\r",
      "\r\n",
      "131256,\"Feuer, Eis & Dosenbier (2002)\",Comedy\r",
      "\r\n",
      "131258,The Pirates (2014),Adventure\r",
      "\r\n",
      "131260,Rentun Ruusu (2001),(no genres listed)\r",
      "\r\n",
      "131262,Innocence (2014),Adventure|Fantasy|Horror\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!tail -10 /tmp/movielens/data/ml-20m/movies.csv "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "27279 /tmp/movielens/data/ml-20m/movies.csv\r\n"
     ]
    }
   ],
   "source": [
    "!wc -l /tmp/movielens/data/ml-20m/movies.csv "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "movieCount = 27279"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "138493,60816,4.5,1259865163\r",
      "\r\n",
      "138493,61160,4.0,1258390537\r",
      "\r\n",
      "138493,65682,4.5,1255816373\r",
      "\r\n",
      "138493,66762,4.5,1255805408\r",
      "\r\n",
      "138493,68319,4.5,1260209720\r",
      "\r\n",
      "138493,68954,4.5,1258126920\r",
      "\r\n",
      "138493,69526,4.5,1259865108\r",
      "\r\n",
      "138493,69644,3.0,1260209457\r",
      "\r\n",
      "138493,70286,5.0,1258126944\r",
      "\r\n",
      "138493,71619,2.5,1255811136\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!tail -10 /tmp/movielens/data/ml-20m/ratings.csv "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "userCount = 138493"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20000264 /tmp/movielens/data/ml-20m/ratings.csv\r\n"
     ]
    }
   ],
   "source": [
    "!wc -l /tmp/movielens/data/ml-20m/ratings.csv "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary\r\n",
      "=======\r\n",
      "\r\n",
      "This dataset (ml-20m) describes 5-star rating and free-text tagging activity from [MovieLens](http://movielens.org), a movie recommendation service. It contains 20000263 ratings and 465564 tag applications across 27278 movies. These data were created by 138493 users between January 09, 1995 and March 31, 2015. This dataset was generated on March 31, 2015, and updated on October 17, 2016 to update links.csv and add genome-* files.\r\n",
      "\r\n",
      "Users were selected at random for inclusion. All selected users had rated at least 20 movies. No demographic information is included. Each user is represented by an id, and no other information is provided.\r\n",
      "\r\n",
      "The data are contained in six files, `genome-scores.csv`, `genome-tags.csv`, `links.csv`, `movies.csv`, `ratings.csv` and `tags.csv`. More details about the contents and use of all these files follows.\r\n",
      "\r\n",
      "This and other GroupLens data sets are publicly available for download at <http://grouplens.org/datasets/>.\r\n",
      "\r\n",
      "\r\n",
      "Usage License\r\n",
      "=============\r\n",
      "\r\n",
      "Neither the University of Minnesota nor any of the researchers involved can guarantee the correctness of the data, its suitability for any particular purpose, or the validity of results based on the use of the data set. The data set may be used for any research purposes under the following conditions:\r\n",
      "\r\n",
      "* The user may not state or imply any endorsement from the University of Minnesota or the GroupLens Research Group.\r\n",
      "* The user must acknowledge the use of the data set in publications resulting from the use of the data set (see below for citation information).\r\n",
      "* The user may not redistribute the data without separate permission.\r\n",
      "* The user may not use this information for any commercial or revenue-bearing purposes without first obtaining permission from a faculty member of the GroupLens Research Project at the University of Minnesota.\r\n",
      "* The executable software scripts are provided \"as is\" without warranty of any kind, either expressed or implied, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. The entire risk as to the quality and performance of them is with you. Should the program prove defective, you assume the cost of all necessary servicing, repair or correction.\r\n",
      "\r\n",
      "In no event shall the University of Minnesota, its affiliates or employees be liable to you for any damages arising out of the use or inability to use these programs (including but not limited to loss of data or data being rendered inaccurate).\r\n",
      "\r\n",
      "If you have any further questions or comments, please email <grouplens-info@umn.edu>\r\n",
      "\r\n",
      "\r\n",
      "Citation\r\n",
      "========\r\n",
      "\r\n",
      "To acknowledge use of the dataset in publications, please cite the following paper:\r\n",
      "\r\n",
      "> F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 4, Article 19 (December 2015), 19 pages. DOI=<http://dx.doi.org/10.1145/2827872>\r\n",
      "\r\n",
      "\r\n",
      "Further Information About GroupLens\r\n",
      "===================================\r\n",
      "\r\n",
      "GroupLens is a research group in the Department of Computer Science and Engineering at the University of Minnesota. Since its inception in 1992, GroupLens's research projects have explored a variety of fields including:\r\n",
      "\r\n",
      "* recommender systems\r\n",
      "* online communities\r\n",
      "* mobile and ubiquitious technologies\r\n",
      "* digital libraries\r\n",
      "* local geographic information systems\r\n",
      "\r\n",
      "GroupLens Research operates a movie recommender based on collaborative filtering, MovieLens, which is the source of these data. We encourage you to visit <http://movielens.org> to try it out! If you have exciting ideas for experimental work to conduct on MovieLens, send us an email at <grouplens-info@cs.umn.edu> - we are always interested in working with external collaborators.\r\n",
      "\r\n",
      "\r\n",
      "Content and Use of Files\r\n",
      "========================\r\n",
      "\r\n",
      "Verifying the Dataset Contents\r\n",
      "------------------------------\r\n",
      "\r\n",
      "We encourage you to verify that the dataset you have on your computer is identical to the ones hosted at [grouplens.org](http://grouplens.org).  This is an important step if you downloaded the dataset from a location other than [grouplens.org](http://grouplens.org), or if you wish to publish research results based on analysis of the MovieLens dataset.\r\n",
      "\r\n",
      "We provide a [MD5 checksum](http://en.wikipedia.org/wiki/Md5sum) with the same name as the downloadable `.zip` file, but with a `.md5` file extension. To verify the dataset:\r\n",
      "\r\n",
      "    # on linux\r\n",
      "    md5sum ml-20m.zip; cat ml-20m.zip.md5\r\n",
      "\r\n",
      "    # on OSX\r\n",
      "    md5 ml-20m.zip; cat ml-20m.zip.md5\r\n",
      "\r\n",
      "    # windows users can download a tool from Microsoft (or elsewhere) that verifies MD5 checksums\r\n",
      "\r\n",
      "Check that the two lines of output contain the same hash value.\r\n",
      "\r\n",
      "Formatting and Encoding\r\n",
      "-----------------------\r\n",
      "\r\n",
      "The dataset files are written as [comma-separated values](http://en.wikipedia.org/wiki/Comma-separated_values) files with a single header row. Columns that contain commas (`,`) are escaped using double-quotes (`\"`). These files are encoded as UTF-8. If accented characters in movie titles or tag values (e.g. Misérables, Les (1995)) display incorrectly, make sure that any program reading the data, such as a text editor, terminal, or script, is configured for UTF-8.\r\n",
      "\r\n",
      "User Ids\r\n",
      "--------\r\n",
      "\r\n",
      "MovieLens users were selected at random for inclusion. Their ids have been anonymized. User ids are consistent between `ratings.csv` and `tags.csv` (i.e., the same id refers to the same user across the two files).\r\n",
      "\r\n",
      "Movie Ids\r\n",
      "---------\r\n",
      "\r\n",
      "Only movies with at least one rating or tag are included in the dataset. These movie ids are consistent with those used on the MovieLens web site (e.g., id `1` corresponds to the URL <https://movielens.org/movies/1>). Movie ids are consistent between `ratings.csv`, `tags.csv`, `movies.csv`, and `links.csv` (i.e., the same id refers to the same movie across these four data files).\r\n",
      "\r\n",
      "\r\n",
      "Ratings Data File Structure (ratings.csv)\r\n",
      "-----------------------------------------\r\n",
      "\r\n",
      "All ratings are contained in the file `ratings.csv`. Each line of this file after the header row represents one rating of one movie by one user, and has the following format:\r\n",
      "\r\n",
      "    userId,movieId,rating,timestamp\r\n",
      "\r\n",
      "The lines within this file are ordered first by userId, then, within user, by movieId.\r\n",
      "\r\n",
      "Ratings are made on a 5-star scale, with half-star increments (0.5 stars - 5.0 stars).\r\n",
      "\r\n",
      "Timestamps represent seconds since midnight Coordinated Universal Time (UTC) of January 1, 1970.\r\n",
      "\r\n",
      "Tags Data File Structure (tags.csv)\r\n",
      "-----------------------------------\r\n",
      "\r\n",
      "All tags are contained in the file `tags.csv`. Each line of this file after the header row represents one tag applied to one movie by one user, and has the following format:\r\n",
      "\r\n",
      "    userId,movieId,tag,timestamp\r\n",
      "\r\n",
      "The lines within this file are ordered first by userId, then, within user, by movieId.\r\n",
      "\r\n",
      "Tags are user-generated metadata about movies. Each tag is typically a single word or short phrase. The meaning, value, and purpose of a particular tag is determined by each user.\r\n",
      "\r\n",
      "Timestamps represent seconds since midnight Coordinated Universal Time (UTC) of January 1, 1970.\r\n",
      "\r\n",
      "Movies Data File Structure (movies.csv)\r\n",
      "---------------------------------------\r\n",
      "\r\n",
      "Movie information is contained in the file `movies.csv`. Each line of this file after the header row represents one movie, and has the following format:\r\n",
      "\r\n",
      "    movieId,title,genres\r\n",
      "\r\n",
      "Movie titles are entered manually or imported from <https://www.themoviedb.org/>, and include the year of release in parentheses. Errors and inconsistencies may exist in these titles.\r\n",
      "\r\n",
      "Genres are a pipe-separated list, and are selected from the following:\r\n",
      "\r\n",
      "* Action\r\n",
      "* Adventure\r\n",
      "* Animation\r\n",
      "* Children's\r\n",
      "* Comedy\r\n",
      "* Crime\r\n",
      "* Documentary\r\n",
      "* Drama\r\n",
      "* Fantasy\r\n",
      "* Film-Noir\r\n",
      "* Horror\r\n",
      "* Musical\r\n",
      "* Mystery\r\n",
      "* Romance\r\n",
      "* Sci-Fi\r\n",
      "* Thriller\r\n",
      "* War\r\n",
      "* Western\r\n",
      "* (no genres listed)\r\n",
      "\r\n",
      "Links Data File Structure (links.csv)\r\n",
      "---------------------------------------\r\n",
      "\r\n",
      "Identifiers that can be used to link to other sources of movie data are contained in the file `links.csv`. Each line of this file after the header row represents one movie, and has the following format:\r\n",
      "\r\n",
      "    movieId,imdbId,tmdbId\r\n",
      "\r\n",
      "movieId is an identifier for movies used by <https://movielens.org>. E.g., the movie Toy Story has the link <https://movielens.org/movies/1>.\r\n",
      "\r\n",
      "imdbId is an identifier for movies used by <http://www.imdb.com>. E.g., the movie Toy Story has the link <http://www.imdb.com/title/tt0114709/>.\r\n",
      "\r\n",
      "tmdbId is an identifier for movies used by <https://www.themoviedb.org>. E.g., the movie Toy Story has the link <https://www.themoviedb.org/movie/862>.\r\n",
      "\r\n",
      "Use of the resources listed above is subject to the terms of each provider.\r\n",
      "\r\n",
      "Tag Genome (genome-scores.csv and genome-tags.csv)\r\n",
      "-------------------------------------------------\r\n",
      "\r\n",
      "This data set includes a current copy of the Tag Genome.\r\n",
      "\r\n",
      "[genome-paper]: http://files.grouplens.org/papers/tag_genome.pdf\r\n",
      "\r\n",
      "The tag genome is a data structure that contains tag relevance scores for movies.  The structure is a dense matrix: each movie in the genome has a value for *every* tag in the genome.\r\n",
      "\r\n",
      "As described in [this article][genome-paper], the tag genome encodes how strongly movies exhibit particular properties represented by tags (atmospheric, thought-provoking, realistic, etc.). The tag genome was computed using a machine learning algorithm on user-contributed content including tags, ratings, and textual reviews.\r\n",
      "\r\n",
      "The genome is split into two files.  The file `genome-scores.csv` contains movie-tag relevance data in the following format:\r\n",
      "\r\n",
      "    movieId,tagId,relevance\r\n",
      "\r\n",
      "The second file, `genome-tags.csv`, provides the tag descriptions for the tag IDs in the genome file, in the following format:\r\n",
      "\r\n",
      "    tagId,tag\r\n",
      "\r\n",
      "The `tagId` values are generated when the data set is exported, so they may vary from version to version of the MovieLens data sets.\r\n",
      "\r\n",
      "Cross-Validation\r\n",
      "----------------\r\n",
      "\r\n",
      "Prior versions of the MovieLens dataset included either pre-computed cross-folds or scripts to perform this computation. We no longer bundle either of these features with the dataset, since most modern toolkits provide this as a built-in feature. If you wish to learn about standard approaches to cross-fold computation in the context of recommender systems evaluation, see [LensKit](http://lenskit.org) for tools, documentation, and open-source code examples.\r\n"
     ]
    }
   ],
   "source": [
    "!cat /tmp/movielens/data/ml-20m/README.txt "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's shuffle input data. We remove the first line which includes column headers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/SageMaker/pilho-lab/ml-100k\n",
      "343\t25\t2\t876402814\n",
      "659\t212\t4\t891387227\n",
      "858\t515\t4\t880932911\n",
      "387\t196\t2\t886484012\n",
      "854\t762\t2\t882812905\n",
      "421\t466\t4\t892241459\n",
      "436\t895\t4\t887768717\n",
      "605\t462\t5\t881016176\n",
      "346\t276\t1\t874950029\n",
      "40\t294\t4\t889041671\n"
     ]
    }
   ],
   "source": [
    "%cd /tmp/movielens/ml-20m\n",
    "!tail -n +2 ratings.csv | shuf -o ratings.shuffled.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "94888,1278,4.5,1394333284\r",
      "\r\n",
      "114817,412,3.5,1106944659\r",
      "\r\n",
      "25180,69757,3.0,1264316440\r",
      "\r\n",
      "18566,141,4.0,861926128\r",
      "\r\n",
      "137648,2542,5.0,973774357\r",
      "\r\n",
      "104166,4973,4.0,1358826903\r",
      "\r\n",
      "36554,64614,4.5,1283328237\r",
      "\r\n",
      "56048,44195,4.5,1275935317\r",
      "\r\n",
      "26460,5312,2.0,1250528982\r",
      "\r\n",
      "12954,5377,3.5,1391219929\r",
      "\r\n"
     ]
    }
   ],
   "source": [
    "!head -10 /tmp/movielens/data/ml-20m/ratings.shuffled.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's separate the training set and the test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "trainingDataRatio = 0.7\n",
    "totalSampleLength = 20000263\n",
    "trainingDataLength = int(totalSampleLength * trainingDataRatio)\n",
    "testingDataLength = totalSampleLength - trainingDataLength"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(14000184, 6000079)\n"
     ]
    }
   ],
   "source": [
    "print(trainingDataLength, testingDataLength)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "!head -14000184 /tmp/movielens/data/ml-20m/ratings.shuffled.csv > /tmp/movielens/data/ml-20m/ratings.train.shuffled.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "!head -6000079 /tmp/movielens/data/ml-20m/ratings.shuffled.csv > /tmp/movielens/data/ml-20m/ratings.test.shuffled.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Build training set and test set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "nbUsers=138493\n",
    "nbMovies=131262\n",
    "nbFeatures=nbUsers+nbMovies\n",
    "\n",
    "nbRatingsTrain=trainingDataLength\n",
    "nbRatingsTest=testingDataLength"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "traingDataSet = '/tmp/movielens/data/ml-20m/ratings.train.shuffled.csv'\n",
    "testingDataSet = '/tmp/movielens/data/ml-20m/ratings.test.shuffled.csv'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For each user, build a list of rated movies.\n",
    "# We'd need this to add random negative samples.\n",
    "moviesByUser = {}\n",
    "for userId in range(nbUsers):\n",
    "    moviesByUser[str(userId)]=[]\n",
    " \n",
    "with open(traingDataSet,'r') as f:\n",
    "    samples=csv.reader(f,delimiter=',')\n",
    "    for userId,movieId,rating,timestamp in samples:\n",
    "        moviesByUser[str(int(userId)-1)].append(int(movieId)-1) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def loadDataset(filename, lines, columns):\n",
    "    # Features are one-hot encoded in a sparse matrix\n",
    "    X = lil_matrix((lines, columns)).astype('float32')\n",
    "    # Labels are stored in a vector\n",
    "    Y = []\n",
    "    line=0\n",
    "    with open(filename,'r') as f:\n",
    "        samples=csv.reader(f,delimiter=',')\n",
    "        for userId,movieId,rating,timestamp in samples:\n",
    "            X[line,int(userId)-1] = 1\n",
    "            X[line,int(nbUsers)+int(movieId)-1] = 1\n",
    "            # Note that this big rating data has a float review score\n",
    "            if float(rating) >= 4.0:\n",
    "                Y.append(1)\n",
    "            else:\n",
    "                Y.append(0)\n",
    "            line=line+1\n",
    "            \n",
    "    Y=np.array(Y).astype('float32')\n",
    "    return X,Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, Y_train = loadDataset(traingDataSet, nbRatingsTrain, nbFeatures)\n",
    "X_test, Y_test = loadDataset(testingDataSet,nbRatingsTest,nbFeatures)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  (0, 44951)\t1.0\n",
      "  (0, 172828)\t1.0\n"
     ]
    }
   ],
   "source": [
    "print(X_train[1000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(14000184, 269755)\n",
      "(14000184,)\n",
      "Training labels: 6996026 zeros, 7004158 ones\n",
      "(6000079, 269755)\n",
      "(6000079,)\n",
      "Test labels: 2998296 zeros, 3001783 ones\n"
     ]
    }
   ],
   "source": [
    "print(X_train.shape)\n",
    "print(Y_train.shape)\n",
    "assert X_train.shape == (nbRatingsTrain, nbFeatures)\n",
    "assert Y_train.shape == (nbRatingsTrain, )\n",
    "zero_labels = np.count_nonzero(Y_train)\n",
    "print(\"Training labels: %d zeros, %d ones\" % (zero_labels, nbRatingsTrain-zero_labels))\n",
    "\n",
    "print(X_test.shape)\n",
    "print(Y_test.shape)\n",
    "assert X_test.shape  == (nbRatingsTest, nbFeatures)\n",
    "assert Y_test.shape  == (nbRatingsTest, )\n",
    "zero_labels = np.count_nonzero(Y_test)\n",
    "print(\"Test labels: %d zeros, %d ones\" % (zero_labels, nbRatingsTest-zero_labels))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Convert to protobuf and save to S3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_key      = 'train.protobuf'\n",
    "train_prefix   = '{}/{}'.format(prefix, 'train3')\n",
    "\n",
    "test_key       = 'test.protobuf'\n",
    "test_prefix    = '{}/{}'.format(prefix, 'test3')\n",
    "\n",
    "output_prefix  = 's3://{}/{}/output'.format(bucket, prefix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def writeDatasetToProtobuf(X, Y, bucket, prefix, key):\n",
    "    buf = io.BytesIO()\n",
    "    smac.write_spmatrix_to_sparse_tensor(buf, X, Y)\n",
    "    buf.seek(0)\n",
    "    obj = '{}/{}'.format(prefix, key)\n",
    "    boto3.resource('s3').Bucket(bucket).Object(obj).upload_fileobj(buf)\n",
    "    return 's3://{}/{}'.format(bucket,obj)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = writeDatasetToProtobuf(X_train, Y_train, bucket, train_prefix, train_key)    \n",
    "test_data  = writeDatasetToProtobuf(X_test, Y_test, bucket, test_prefix, test_key)    \n",
    "  \n",
    "print(train_data)\n",
    "print(test_data)\n",
    "print('Output: {}'.format(output_prefix))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run training job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data = 's3://pilho-sagemaker-ai-workshop/sagemaker/fm-movielens/train3/train.protobuf'\n",
    "test_data = 's3://pilho-sagemaker-ai-workshop/sagemaker/fm-movielens/test3/test.protobuf'\n",
    "output_prefix = 's3://pilho-sagemaker-ai-workshop/sagemaker/fm-movielens/output'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "from sagemaker.amazon.amazon_estimator import get_image_uri\n",
    "container = get_image_uri(boto3.Session().region_name, 'factorization-machines')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:sagemaker:Creating training-job with name: factorization-machines-2018-08-29-13-05-40-942\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      ".....................\n",
      "\u001b[33mDocker entrypoint called with argument(s): train\u001b[0m\n",
      "\u001b[32mDocker entrypoint called with argument(s): train\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:06 INFO 140667034941248] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/resources/default-conf.json: {u'factors_lr': u'0.0001', u'linear_init_sigma': u'0.01', u'epochs': 1, u'_wd': u'1.0', u'_num_kv_servers': u'auto', u'use_bias': u'true', u'factors_init_sigma': u'0.001', u'_log_level': u'info', u'bias_init_method': u'normal', u'linear_init_method': u'normal', u'linear_lr': u'0.001', u'factors_init_method': u'normal', u'_tuning_objective_metric': u'', u'bias_wd': u'0.01', u'use_linear': u'true', u'bias_lr': u'0.1', u'mini_batch_size': u'1000', u'_use_full_symbolic': u'true', u'batch_metrics_publish_interval': u'500', u'bias_init_sigma': u'0.01', u'_num_gpus': u'auto', u'_data_format': u'record', u'factors_wd': u'0.00001', u'linear_wd': u'0.001', u'_kvstore': u'auto', u'_learning_rate': u'1.0', u'_optimizer': u'adam'}\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:06 INFO 140667034941248] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'epochs': u'100', u'feature_dim': u'269755', u'mini_batch_size': u'1000', u'predictor_type': u'binary_classifier', u'num_factors': u'64'}\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:06 INFO 140667034941248] Final configuration: {u'factors_lr': u'0.0001', u'linear_init_sigma': u'0.01', u'epochs': u'100', u'feature_dim': u'269755', u'num_factors': u'64', u'_wd': u'1.0', u'_num_kv_servers': u'auto', u'use_bias': u'true', u'factors_init_sigma': u'0.001', u'_log_level': u'info', u'bias_init_method': u'normal', u'linear_init_method': u'normal', u'linear_lr': u'0.001', u'factors_init_method': u'normal', u'_tuning_objective_metric': u'', u'bias_wd': u'0.01', u'use_linear': u'true', u'bias_lr': u'0.1', u'mini_batch_size': u'1000', u'_use_full_symbolic': u'true', u'batch_metrics_publish_interval': u'500', u'predictor_type': u'binary_classifier', u'bias_init_sigma': u'0.01', u'_num_gpus': u'auto', u'_data_format': u'record', u'factors_wd': u'0.00001', u'linear_wd': u'0.001', u'_kvstore': u'auto', u'_learning_rate': u'1.0', u'_optimizer': u'adam'}\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:06 WARNING 140667034941248] Loggers have already been setup.\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:07 INFO 140658308478784] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/resources/default-conf.json: {u'factors_lr': u'0.0001', u'linear_init_sigma': u'0.01', u'epochs': 1, u'_wd': u'1.0', u'_num_kv_servers': u'auto', u'use_bias': u'true', u'factors_init_sigma': u'0.001', u'_log_level': u'info', u'bias_init_method': u'normal', u'linear_init_method': u'normal', u'linear_lr': u'0.001', u'factors_init_method': u'normal', u'_tuning_objective_metric': u'', u'bias_wd': u'0.01', u'use_linear': u'true', u'bias_lr': u'0.1', u'mini_batch_size': u'1000', u'_use_full_symbolic': u'true', u'batch_metrics_publish_interval': u'500', u'bias_init_sigma': u'0.01', u'_num_gpus': u'auto', u'_data_format': u'record', u'factors_wd': u'0.00001', u'linear_wd': u'0.001', u'_kvstore': u'auto', u'_learning_rate': u'1.0', u'_optimizer': u'adam'}\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:07 INFO 140658308478784] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'epochs': u'100', u'feature_dim': u'269755', u'mini_batch_size': u'1000', u'predictor_type': u'binary_classifier', u'num_factors': u'64'}\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:07 INFO 140658308478784] Final configuration: {u'factors_lr': u'0.0001', u'linear_init_sigma': u'0.01', u'epochs': u'100', u'feature_dim': u'269755', u'num_factors': u'64', u'_wd': u'1.0', u'_num_kv_servers': u'auto', u'use_bias': u'true', u'factors_init_sigma': u'0.001', u'_log_level': u'info', u'bias_init_method': u'normal', u'linear_init_method': u'normal', u'linear_lr': u'0.001', u'factors_init_method': u'normal', u'_tuning_objective_metric': u'', u'bias_wd': u'0.01', u'use_linear': u'true', u'bias_lr': u'0.1', u'mini_batch_size': u'1000', u'_use_full_symbolic': u'true', u'batch_metrics_publish_interval': u'500', u'predictor_type': u'binary_classifier', u'bias_init_sigma': u'0.01', u'_num_gpus': u'auto', u'_data_format': u'record', u'factors_wd': u'0.00001', u'linear_wd': u'0.001', u'_kvstore': u'auto', u'_learning_rate': u'1.0', u'_optimizer': u'adam'}\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:07 WARNING 140658308478784] Loggers have already been setup.\u001b[0m\n",
      "\u001b[34mDocker entrypoint called with argument(s): train\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:08 INFO 140684726773568] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/resources/default-conf.json: {u'factors_lr': u'0.0001', u'linear_init_sigma': u'0.01', u'epochs': 1, u'_wd': u'1.0', u'_num_kv_servers': u'auto', u'use_bias': u'true', u'factors_init_sigma': u'0.001', u'_log_level': u'info', u'bias_init_method': u'normal', u'linear_init_method': u'normal', u'linear_lr': u'0.001', u'factors_init_method': u'normal', u'_tuning_objective_metric': u'', u'bias_wd': u'0.01', u'use_linear': u'true', u'bias_lr': u'0.1', u'mini_batch_size': u'1000', u'_use_full_symbolic': u'true', u'batch_metrics_publish_interval': u'500', u'bias_init_sigma': u'0.01', u'_num_gpus': u'auto', u'_data_format': u'record', u'factors_wd': u'0.00001', u'linear_wd': u'0.001', u'_kvstore': u'auto', u'_learning_rate': u'1.0', u'_optimizer': u'adam'}\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:08 INFO 140684726773568] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'epochs': u'100', u'feature_dim': u'269755', u'mini_batch_size': u'1000', u'predictor_type': u'binary_classifier', u'num_factors': u'64'}\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:08 INFO 140684726773568] Final configuration: {u'factors_lr': u'0.0001', u'linear_init_sigma': u'0.01', u'epochs': u'100', u'feature_dim': u'269755', u'num_factors': u'64', u'_wd': u'1.0', u'_num_kv_servers': u'auto', u'use_bias': u'true', u'factors_init_sigma': u'0.001', u'_log_level': u'info', u'bias_init_method': u'normal', u'linear_init_method': u'normal', u'linear_lr': u'0.001', u'factors_init_method': u'normal', u'_tuning_objective_metric': u'', u'bias_wd': u'0.01', u'use_linear': u'true', u'bias_lr': u'0.1', u'mini_batch_size': u'1000', u'_use_full_symbolic': u'true', u'batch_metrics_publish_interval': u'500', u'predictor_type': u'binary_classifier', u'bias_init_sigma': u'0.01', u'_num_gpus': u'auto', u'_data_format': u'record', u'factors_wd': u'0.00001', u'linear_wd': u'0.001', u'_kvstore': u'auto', u'_learning_rate': u'1.0', u'_optimizer': u'adam'}\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:08 WARNING 140684726773568] Loggers have already been setup.\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:11 INFO 140658308478784] Launching parameter server for role server\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:11 INFO 140658308478784] {'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'ethwe', 'SHLVL': '1', 'AWS_REGION': 'us-east-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'TRAINING_JOB_NAME': 'factorization-machines-2018-08-29-13-05-40-942', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '36', 'HOSTNAME': 'aws', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/52cea8f3-69eb-495d-b631-e7c81ca1c849', 'PWD': '/'}\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:11 INFO 140658308478784] envs={'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'DMLC_NUM_WORKER': '4', 'DMLC_PS_ROOT_PORT': '9000', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'ethwe', 'SHLVL': '1', 'DMLC_PS_ROOT_URI': '10.44.0.3', 'AWS_REGION': 'us-east-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'TRAINING_JOB_NAME': 'factorization-machines-2018-08-29-13-05-40-942', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '36', 'HOSTNAME': 'aws', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/52cea8f3-69eb-495d-b631-e7c81ca1c849', 'DMLC_ROLE': 'server', 'PWD': '/', 'DMLC_NUM_SERVER': '4'}\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:11 INFO 140658308478784] Environment: {'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'DMLC_PS_ROOT_PORT': '9000', 'DMLC_NUM_WORKER': '4', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'ethwe', 'SHLVL': '1', 'DMLC_PS_ROOT_URI': '10.44.0.3', 'AWS_REGION': 'us-east-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'TRAINING_JOB_NAME': 'factorization-machines-2018-08-29-13-05-40-942', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '36', 'HOSTNAME': 'aws', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/52cea8f3-69eb-495d-b631-e7c81ca1c849', 'DMLC_ROLE': 'worker', 'PWD': '/', 'DMLC_NUM_SERVER': '4'}\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:11 INFO 140658308478784] Using default worker.\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:11 INFO 140658308478784] nvidia-smi took: 0.0251941680908 secs to identify 0 gpus\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:11 INFO 140658308478784] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:11 INFO 140658308478784] [Sparse network] Building a sparse network.\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:11 INFO 140658308478784] Create Store: dist_async\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:11 INFO 140667034941248] Launching parameter server for role server\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:11 INFO 140667034941248] {'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'ethwe', 'SHLVL': '1', 'AWS_REGION': 'us-east-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'TRAINING_JOB_NAME': 'factorization-machines-2018-08-29-13-05-40-942', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '36', 'HOSTNAME': 'aws', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/9c698c28-8efd-4547-90e6-16483f78b760', 'PWD': '/'}\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:11 INFO 140667034941248] envs={'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'DMLC_NUM_WORKER': '4', 'DMLC_PS_ROOT_PORT': '9000', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'ethwe', 'SHLVL': '1', 'DMLC_PS_ROOT_URI': '10.44.0.3', 'AWS_REGION': 'us-east-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'TRAINING_JOB_NAME': 'factorization-machines-2018-08-29-13-05-40-942', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '36', 'HOSTNAME': 'aws', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/9c698c28-8efd-4547-90e6-16483f78b760', 'DMLC_ROLE': 'server', 'PWD': '/', 'DMLC_NUM_SERVER': '4'}\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:11 INFO 140667034941248] Environment: {'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'DMLC_PS_ROOT_PORT': '9000', 'DMLC_NUM_WORKER': '4', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'ethwe', 'SHLVL': '1', 'DMLC_PS_ROOT_URI': '10.44.0.3', 'AWS_REGION': 'us-east-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'TRAINING_JOB_NAME': 'factorization-machines-2018-08-29-13-05-40-942', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '36', 'HOSTNAME': 'aws', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/9c698c28-8efd-4547-90e6-16483f78b760', 'DMLC_ROLE': 'worker', 'PWD': '/', 'DMLC_NUM_SERVER': '4'}\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:11 INFO 140667034941248] Using default worker.\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:11 INFO 140667034941248] nvidia-smi took: 0.025200843811 secs to identify 0 gpus\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:11 INFO 140667034941248] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:11 INFO 140667034941248] [Sparse network] Building a sparse network.\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:11 INFO 140667034941248] Create Store: dist_async\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:11 INFO 140684726773568] Launching parameter server for role server\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:11 INFO 140684726773568] {'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'ethwe', 'SHLVL': '1', 'AWS_REGION': 'us-east-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'TRAINING_JOB_NAME': 'factorization-machines-2018-08-29-13-05-40-942', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '36', 'HOSTNAME': 'aws', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/f1ea10d9-142e-4c52-a884-4697add8f4a9', 'PWD': '/'}\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:11 INFO 140684726773568] envs={'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'DMLC_NUM_WORKER': '4', 'DMLC_PS_ROOT_PORT': '9000', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'ethwe', 'SHLVL': '1', 'DMLC_PS_ROOT_URI': '10.44.0.3', 'AWS_REGION': 'us-east-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'TRAINING_JOB_NAME': 'factorization-machines-2018-08-29-13-05-40-942', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '36', 'HOSTNAME': 'aws', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/f1ea10d9-142e-4c52-a884-4697add8f4a9', 'DMLC_ROLE': 'server', 'PWD': '/', 'DMLC_NUM_SERVER': '4'}\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:11 INFO 140684726773568] Environment: {'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'DMLC_PS_ROOT_PORT': '9000', 'DMLC_NUM_WORKER': '4', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'ethwe', 'SHLVL': '1', 'DMLC_PS_ROOT_URI': '10.44.0.3', 'AWS_REGION': 'us-east-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'TRAINING_JOB_NAME': 'factorization-machines-2018-08-29-13-05-40-942', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '36', 'HOSTNAME': 'aws', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/f1ea10d9-142e-4c52-a884-4697add8f4a9', 'DMLC_ROLE': 'worker', 'PWD': '/', 'DMLC_NUM_SERVER': '4'}\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:11 INFO 140684726773568] Using default worker.\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:11 INFO 140684726773568] nvidia-smi took: 0.0251929759979 secs to identify 0 gpus\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:11 INFO 140684726773568] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:11 INFO 140684726773568] [Sparse network] Building a sparse network.\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:11 INFO 140684726773568] Create Store: dist_async\u001b[0m\n",
      "\u001b[31mDocker entrypoint called with argument(s): train\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:11 INFO 140009951377216] Reading default configuration from /opt/amazon/lib/python2.7/site-packages/algorithm/resources/default-conf.json: {u'factors_lr': u'0.0001', u'linear_init_sigma': u'0.01', u'epochs': 1, u'_wd': u'1.0', u'_num_kv_servers': u'auto', u'use_bias': u'true', u'factors_init_sigma': u'0.001', u'_log_level': u'info', u'bias_init_method': u'normal', u'linear_init_method': u'normal', u'linear_lr': u'0.001', u'factors_init_method': u'normal', u'_tuning_objective_metric': u'', u'bias_wd': u'0.01', u'use_linear': u'true', u'bias_lr': u'0.1', u'mini_batch_size': u'1000', u'_use_full_symbolic': u'true', u'batch_metrics_publish_interval': u'500', u'bias_init_sigma': u'0.01', u'_num_gpus': u'auto', u'_data_format': u'record', u'factors_wd': u'0.00001', u'linear_wd': u'0.001', u'_kvstore': u'auto', u'_learning_rate': u'1.0', u'_optimizer': u'adam'}\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:11 INFO 140009951377216] Reading provided configuration from /opt/ml/input/config/hyperparameters.json: {u'epochs': u'100', u'feature_dim': u'269755', u'mini_batch_size': u'1000', u'predictor_type': u'binary_classifier', u'num_factors': u'64'}\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:11 INFO 140009951377216] Final configuration: {u'factors_lr': u'0.0001', u'linear_init_sigma': u'0.01', u'epochs': u'100', u'feature_dim': u'269755', u'num_factors': u'64', u'_wd': u'1.0', u'_num_kv_servers': u'auto', u'use_bias': u'true', u'factors_init_sigma': u'0.001', u'_log_level': u'info', u'bias_init_method': u'normal', u'linear_init_method': u'normal', u'linear_lr': u'0.001', u'factors_init_method': u'normal', u'_tuning_objective_metric': u'', u'bias_wd': u'0.01', u'use_linear': u'true', u'bias_lr': u'0.1', u'mini_batch_size': u'1000', u'_use_full_symbolic': u'true', u'batch_metrics_publish_interval': u'500', u'predictor_type': u'binary_classifier', u'bias_init_sigma': u'0.01', u'_num_gpus': u'auto', u'_data_format': u'record', u'factors_wd': u'0.00001', u'linear_wd': u'0.001', u'_kvstore': u'auto', u'_learning_rate': u'1.0', u'_optimizer': u'adam'}\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:11 WARNING 140009951377216] Loggers have already been setup.\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:11 INFO 140009951377216] Launching parameter server for role scheduler\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:11 INFO 140009951377216] {'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'ethwe', 'SHLVL': '1', 'AWS_REGION': 'us-east-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'TRAINING_JOB_NAME': 'factorization-machines-2018-08-29-13-05-40-942', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '36', 'HOSTNAME': 'aws', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/b12b512f-f292-421b-93ad-a78cf9235ef7', 'PWD': '/'}\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:11 INFO 140009951377216] envs={'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'DMLC_NUM_WORKER': '4', 'DMLC_PS_ROOT_PORT': '9000', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'ethwe', 'SHLVL': '1', 'DMLC_PS_ROOT_URI': '10.44.0.3', 'AWS_REGION': 'us-east-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'TRAINING_JOB_NAME': 'factorization-machines-2018-08-29-13-05-40-942', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '36', 'HOSTNAME': 'aws', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/b12b512f-f292-421b-93ad-a78cf9235ef7', 'DMLC_ROLE': 'scheduler', 'PWD': '/', 'DMLC_NUM_SERVER': '4'}\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:11 INFO 140009951377216] Launching parameter server for role server\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:11 INFO 140009951377216] {'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'ethwe', 'SHLVL': '1', 'AWS_REGION': 'us-east-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'TRAINING_JOB_NAME': 'factorization-machines-2018-08-29-13-05-40-942', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '36', 'HOSTNAME': 'aws', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/b12b512f-f292-421b-93ad-a78cf9235ef7', 'PWD': '/'}\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:11 INFO 140009951377216] envs={'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'DMLC_NUM_WORKER': '4', 'DMLC_PS_ROOT_PORT': '9000', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'ethwe', 'SHLVL': '1', 'DMLC_PS_ROOT_URI': '10.44.0.3', 'AWS_REGION': 'us-east-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'TRAINING_JOB_NAME': 'factorization-machines-2018-08-29-13-05-40-942', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '36', 'HOSTNAME': 'aws', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/b12b512f-f292-421b-93ad-a78cf9235ef7', 'DMLC_ROLE': 'server', 'PWD': '/', 'DMLC_NUM_SERVER': '4'}\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:11 INFO 140009951377216] Environment: {'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION_VERSION': '2', 'DMLC_PS_ROOT_PORT': '9000', 'DMLC_NUM_WORKER': '4', 'SAGEMAKER_HTTP_PORT': '8080', 'HOME': '/root', 'PYTHONUNBUFFERED': 'TRUE', 'CANONICAL_ENVROOT': '/opt/amazon', 'LD_LIBRARY_PATH': '/usr/local/nvidia/lib64:/opt/amazon/lib', 'LANG': 'en_US.utf8', 'DMLC_INTERFACE': 'ethwe', 'SHLVL': '1', 'DMLC_PS_ROOT_URI': '10.44.0.3', 'AWS_REGION': 'us-east-1', 'NVIDIA_VISIBLE_DEVICES': 'all', 'TRAINING_JOB_NAME': 'factorization-machines-2018-08-29-13-05-40-942', 'PATH': '/opt/amazon/bin:/usr/local/nvidia/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/amazon/bin:/opt/amazon/bin', 'PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION': 'cpp', 'ENVROOT': '/opt/amazon', 'SAGEMAKER_DATA_PATH': '/opt/ml', 'NVIDIA_DRIVER_CAPABILITIES': 'compute,utility', 'NVIDIA_REQUIRE_CUDA': 'cuda>=9.0', 'OMP_NUM_THREADS': '36', 'HOSTNAME': 'aws', 'AWS_CONTAINER_CREDENTIALS_RELATIVE_URI': '/v2/credentials/b12b512f-f292-421b-93ad-a78cf9235ef7', 'DMLC_ROLE': 'worker', 'PWD': '/', 'DMLC_NUM_SERVER': '4'}\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:11 INFO 140009951377216] Using default worker.\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:11 INFO 140009951377216] nvidia-smi took: 0.0251548290253 secs to identify 0 gpus\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:11 INFO 140009951377216] Number of GPUs being used: 0\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:11 INFO 140009951377216] [Sparse network] Building a sparse network.\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:11 INFO 140009951377216] Create Store: dist_async\u001b[0m\n",
      "\u001b[32m#metrics {\"Metrics\": {\"initialize.time\": {\"count\": 1, \"max\": 1991.9040203094482, \"sum\": 1991.9040203094482, \"min\": 1991.9040203094482}}, \"EndTime\": 1535548153.64554, \"Dimensions\": {\"Host\": \"algo-2\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535548151.622596}\n",
      "\u001b[0m\n",
      "\u001b[32m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}, \"Total Records Seen\": {\"count\": 1, \"max\": 1000, \"sum\": 1000.0, \"min\": 1000}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 1000, \"sum\": 1000.0, \"min\": 1000}, \"Reset Count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1535548153.645707, \"Dimensions\": {\"Host\": \"algo-2\", \"Meta\": \"init_train_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535548153.645678}\n",
      "\u001b[0m\n",
      "\u001b[32m[13:09:13] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.1.x.200686.0/RHEL5_64/generic-flavor/src/src/kvstore/./kvstore_local.h:280: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use row_sparse_pull with row_ids.\u001b[0m\n",
      "\u001b[33m#metrics {\"Metrics\": {\"initialize.time\": {\"count\": 1, \"max\": 2415.166139602661, \"sum\": 2415.166139602661, \"min\": 2415.166139602661}}, \"EndTime\": 1535548153.644268, \"Dimensions\": {\"Host\": \"algo-3\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535548151.196465}\n",
      "\u001b[0m\n",
      "\u001b[33m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}, \"Total Records Seen\": {\"count\": 1, \"max\": 1000, \"sum\": 1000.0, \"min\": 1000}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 1000, \"sum\": 1000.0, \"min\": 1000}, \"Reset Count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1535548153.644459, \"Dimensions\": {\"Host\": \"algo-3\", \"Meta\": \"init_train_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535548153.644428}\n",
      "\u001b[0m\n",
      "\u001b[33m[13:09:13] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.1.x.200686.0/RHEL5_64/generic-flavor/src/src/kvstore/./kvstore_local.h:280: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use row_sparse_pull with row_ids.\u001b[0m\n",
      "\u001b[33m[13:09:13] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.1.x.200686.0/RHEL5_64/generic-flavor/src/src/kvstore/./kvstore_local.h:280: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use row_sparse_pull with row_ids.\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:14 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=0 train binary_classification_accuracy <score>=0.514\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:14 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=0 train binary_classification_cross_entropy <loss>=0.692756835938\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:14 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=0 train binary_f_1.000 <score>=0.667578659371\u001b[0m\n",
      "\u001b[31m#metrics {\"Metrics\": {\"initialize.time\": {\"count\": 1, \"max\": 1832.4999809265137, \"sum\": 1832.4999809265137, \"min\": 1832.4999809265137}}, \"EndTime\": 1535548153.643061, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535548151.783757}\n",
      "\u001b[0m\n",
      "\u001b[31m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}, \"Total Records Seen\": {\"count\": 1, \"max\": 1000, \"sum\": 1000.0, \"min\": 1000}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 1000, \"sum\": 1000.0, \"min\": 1000}, \"Reset Count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1535548153.643232, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"init_train_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535548153.643201}\n",
      "\u001b[0m\n",
      "\u001b[31m[13:09:13] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.1.x.200686.0/RHEL5_64/generic-flavor/src/src/kvstore/./kvstore_local.h:280: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use row_sparse_pull with row_ids.\u001b[0m\n",
      "\u001b[31m[13:09:13] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.1.x.200686.0/RHEL5_64/generic-flavor/src/src/kvstore/./kvstore_local.h:280: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use row_sparse_pull with row_ids.\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:14 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=0 train binary_classification_accuracy <score>=0.514\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:14 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=0 train binary_classification_cross_entropy <loss>=0.692756835938\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:14 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=0 train binary_f_1.000 <score>=0.667578659371\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"initialize.time\": {\"count\": 1, \"max\": 2077.9788494110107, \"sum\": 2077.9788494110107, \"min\": 2077.9788494110107}}, \"EndTime\": 1535548153.645036, \"Dimensions\": {\"Host\": \"algo-4\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535548151.534737}\n",
      "\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 0, \"sum\": 0.0, \"min\": 0}, \"Total Batches Seen\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}, \"Total Records Seen\": {\"count\": 1, \"max\": 1000, \"sum\": 1000.0, \"min\": 1000}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 1000, \"sum\": 1000.0, \"min\": 1000}, \"Reset Count\": {\"count\": 1, \"max\": 1, \"sum\": 1.0, \"min\": 1}}, \"EndTime\": 1535548153.645199, \"Dimensions\": {\"Host\": \"algo-4\", \"Meta\": \"init_train_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535548153.645172}\n",
      "\u001b[0m\n",
      "\u001b[34m[13:09:13] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.1.x.200686.0/RHEL5_64/generic-flavor/src/src/kvstore/./kvstore_local.h:280: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use row_sparse_pull with row_ids.\u001b[0m\n",
      "\u001b[34m[13:09:13] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.1.x.200686.0/RHEL5_64/generic-flavor/src/src/kvstore/./kvstore_local.h:280: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use row_sparse_pull with row_ids.\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=0 train binary_classification_accuracy <score>=0.514\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=0 train binary_classification_cross_entropy <loss>=0.692756835938\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=0 train binary_f_1.000 <score>=0.667578659371\u001b[0m\n",
      "\u001b[32m[13:09:13] /opt/brazil-pkg-cache/packages/AIAlgorithmsMXNet/AIAlgorithmsMXNet-1.1.x.200686.0/RHEL5_64/generic-flavor/src/src/kvstore/./kvstore_local.h:280: Warning: non-default weights detected during kvstore pull. This call has been ignored. Please make sure to use row_sparse_pull with row_ids.\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:14 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=0 train binary_classification_accuracy <score>=0.514\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:14 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=0 train binary_classification_cross_entropy <loss>=0.692756835938\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:14 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=0 train binary_f_1.000 <score>=0.667578659371\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[08/29/2018 13:09:35 INFO 140658308478784] Iter[0] Batch [500]#011Speed: 23141.85 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:35 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=500 train binary_classification_accuracy <score>=0.572219560878\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:35 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=500 train binary_classification_cross_entropy <loss>=0.6760168228\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:35 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=500 train binary_f_1.000 <score>=0.572067520995\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:35 INFO 140667034941248] Iter[0] Batch [500]#011Speed: 23374.13 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:35 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=500 train binary_classification_accuracy <score>=0.574399201597\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:35 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=500 train binary_classification_cross_entropy <loss>=0.674983392224\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:35 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=500 train binary_f_1.000 <score>=0.576025705775\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:36 INFO 140009951377216] Iter[0] Batch [500]#011Speed: 22305.61 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:36 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=500 train binary_classification_accuracy <score>=0.578301397206\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:36 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=500 train binary_classification_cross_entropy <loss>=0.674413203988\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:36 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=500 train binary_f_1.000 <score>=0.579927187989\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:36 INFO 140684726773568] Iter[0] Batch [500]#011Speed: 21937.78 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:36 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=500 train binary_classification_accuracy <score>=0.585600798403\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:36 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=500 train binary_classification_cross_entropy <loss>=0.671282801828\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:36 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=500 train binary_f_1.000 <score>=0.589401904129\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:56 INFO 140667034941248] Iter[0] Batch [1000]#011Speed: 23530.50 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:56 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=1000 train binary_classification_accuracy <score>=0.598341658342\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:56 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=1000 train binary_classification_cross_entropy <loss>=0.666443271206\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:09:56 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=1000 train binary_f_1.000 <score>=0.598303133967\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:57 INFO 140658308478784] Iter[0] Batch [1000]#011Speed: 23028.00 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:57 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=1000 train binary_classification_accuracy <score>=0.598412587413\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:57 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=1000 train binary_classification_cross_entropy <loss>=0.666282310097\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:09:57 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=1000 train binary_f_1.000 <score>=0.596094468252\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:59 INFO 140684726773568] Iter[0] Batch [1000]#011Speed: 22543.85 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:59 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=1000 train binary_classification_accuracy <score>=0.606614385614\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:59 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=1000 train binary_classification_cross_entropy <loss>=0.662916473846\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:09:59 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=1000 train binary_f_1.000 <score>=0.604885116579\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:59 INFO 140009951377216] Iter[0] Batch [1000]#011Speed: 21328.91 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:59 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=1000 train binary_classification_accuracy <score>=0.603997002997\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:59 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=1000 train binary_classification_cross_entropy <loss>=0.664148449878\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:09:59 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=1000 train binary_f_1.000 <score>=0.603709979756\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:10:18 INFO 140658308478784] Iter[0] Batch [1500]#011Speed: 23346.49 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:10:18 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=1500 train binary_classification_accuracy <score>=0.609201199201\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:10:18 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=1500 train binary_classification_cross_entropy <loss>=0.661835419535\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:10:18 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=1500 train binary_f_1.000 <score>=0.607495464969\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:10:19 INFO 140667034941248] Iter[0] Batch [1500]#011Speed: 21736.19 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:10:19 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=1500 train binary_classification_accuracy <score>=0.608634243837\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:10:19 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=1500 train binary_classification_cross_entropy <loss>=0.662042446434\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:10:19 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=1500 train binary_f_1.000 <score>=0.608665008347\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:10:21 INFO 140684726773568] Iter[0] Batch [1500]#011Speed: 22749.22 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:10:21 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=1500 train binary_classification_accuracy <score>=0.617898734177\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:10:21 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=1500 train binary_classification_cross_entropy <loss>=0.658667716395\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:10:21 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=1500 train binary_f_1.000 <score>=0.615414945021\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:10:21 INFO 140009951377216] Iter[0] Batch [1500]#011Speed: 23126.83 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:10:21 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=1500 train binary_classification_accuracy <score>=0.617948034644\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:10:21 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=1500 train binary_classification_cross_entropy <loss>=0.658919385733\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:10:21 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=1500 train binary_f_1.000 <score>=0.61801471029\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:10:41 INFO 140658308478784] Iter[0] Batch [2000]#011Speed: 22166.04 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:10:41 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=2000 train binary_classification_accuracy <score>=0.617898050975\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:10:41 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=2000 train binary_classification_cross_entropy <loss>=0.658792138507\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:10:41 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=2000 train binary_f_1.000 <score>=0.616400075056\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:10:41 INFO 140667034941248] Iter[0] Batch [2000]#011Speed: 22605.13 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:10:41 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=2000 train binary_classification_accuracy <score>=0.618108445777\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:10:41 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=2000 train binary_classification_cross_entropy <loss>=0.658733367935\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:10:41 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=2000 train binary_f_1.000 <score>=0.617455960616\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:10:43 INFO 140684726773568] Iter[0] Batch [2000]#011Speed: 22420.96 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:10:43 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=2000 train binary_classification_accuracy <score>=0.625132433783\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:10:43 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=2000 train binary_classification_cross_entropy <loss>=0.655860027902\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:10:43 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=2000 train binary_f_1.000 <score>=0.62218049468\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:10:45 INFO 140009951377216] Iter[0] Batch [2000]#011Speed: 21271.42 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:10:45 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=2000 train binary_classification_accuracy <score>=0.625797601199\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:10:45 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=2000 train binary_classification_cross_entropy <loss>=0.655547610137\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:10:45 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=2000 train binary_f_1.000 <score>=0.624652049007\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[08/29/2018 13:11:04 INFO 140658308478784] Iter[0] Batch [2500]#011Speed: 21393.92 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:11:04 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=2500 train binary_classification_accuracy <score>=0.623036385446\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:11:04 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=2500 train binary_classification_cross_entropy <loss>=0.656341308447\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:11:04 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=2500 train binary_f_1.000 <score>=0.621729302131\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:11:05 INFO 140667034941248] Iter[0] Batch [2500]#011Speed: 21252.25 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:11:05 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=2500 train binary_classification_accuracy <score>=0.623303878449\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:11:05 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=2500 train binary_classification_cross_entropy <loss>=0.656142100752\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:11:05 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=2500 train binary_f_1.000 <score>=0.622169480459\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:11:07 INFO 140684726773568] Iter[0] Batch [2500]#011Speed: 20818.55 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:11:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=2500 train binary_classification_accuracy <score>=0.629259496202\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:11:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=2500 train binary_classification_cross_entropy <loss>=0.653538687845\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:11:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=2500 train binary_f_1.000 <score>=0.626061356423\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:11:08 INFO 140009951377216] Iter[0] Batch [2500]#011Speed: 21653.72 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:11:08 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=2500 train binary_classification_accuracy <score>=0.630791683327\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:11:08 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=2500 train binary_classification_cross_entropy <loss>=0.65297019842\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:11:08 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=2500 train binary_f_1.000 <score>=0.629624430131\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:11:29 INFO 140667034941248] Iter[0] Batch [3000]#011Speed: 20489.86 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:11:29 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=3000 train binary_classification_accuracy <score>=0.63117694102\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:11:29 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=3000 train binary_classification_cross_entropy <loss>=0.653360150905\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:11:29 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=3000 train binary_f_1.000 <score>=0.630328457987\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:11:29 INFO 140658308478784] Iter[0] Batch [3000]#011Speed: 20507.66 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:11:29 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=3000 train binary_classification_accuracy <score>=0.630186937687\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:11:29 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=3000 train binary_classification_cross_entropy <loss>=0.653794885937\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:11:29 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=3000 train binary_f_1.000 <score>=0.62903572466\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:11:30 INFO 140684726773568] Iter[0] Batch [3000]#011Speed: 21721.82 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:11:30 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=3000 train binary_classification_accuracy <score>=0.637802732423\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:11:30 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=3000 train binary_classification_cross_entropy <loss>=0.650736710118\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:11:30 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=3000 train binary_f_1.000 <score>=0.63515875797\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:11:32 INFO 140009951377216] Iter[0] Batch [3000]#011Speed: 20791.27 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:11:32 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=3000 train binary_classification_accuracy <score>=0.640127957348\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:11:32 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=3000 train binary_classification_cross_entropy <loss>=0.649938912513\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:11:32 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=3000 train binary_f_1.000 <score>=0.638926705842\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:11:52 INFO 140658308478784] Iter[0] Batch [3500]#011Speed: 21547.53 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:11:52 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=3500 train binary_classification_accuracy <score>=0.634326763782\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:11:52 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=3500 train binary_classification_cross_entropy <loss>=0.651410982029\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:11:52 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=3500 train binary_f_1.000 <score>=0.633347557745\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:11:53 INFO 140667034941248] Iter[0] Batch [3500]#011Speed: 21006.56 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:11:53 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=3500 train binary_classification_accuracy <score>=0.635841759497\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:11:53 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=3500 train binary_classification_cross_entropy <loss>=0.650800720145\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:11:53 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=3500 train binary_f_1.000 <score>=0.6346110636\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:11:54 INFO 140009951377216] Iter[0] Batch [3500]#011Speed: 22351.96 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:11:54 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=3500 train binary_classification_accuracy <score>=0.643803199086\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:11:54 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=3500 train binary_classification_cross_entropy <loss>=0.647472156881\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:11:54 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=3500 train binary_f_1.000 <score>=0.642533944434\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:11:54 INFO 140684726773568] Iter[0] Batch [3500]#011Speed: 21013.98 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:11:54 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=3500 train binary_classification_accuracy <score>=0.641808054841\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:11:54 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=3500 train binary_classification_cross_entropy <loss>=0.648324364007\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:11:54 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=3500 train binary_f_1.000 <score>=0.639702666998\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:12:14 INFO 140658308478784] Iter[0] Batch [4000]#011Speed: 22773.93 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:12:14 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=4000 train binary_classification_accuracy <score>=0.639430642339\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:12:14 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=4000 train binary_classification_cross_entropy <loss>=0.648660383763\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:12:14 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=4000 train binary_f_1.000 <score>=0.638814252366\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:12:16 INFO 140684726773568] Iter[0] Batch [4000]#011Speed: 22103.06 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:12:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=4000 train binary_classification_accuracy <score>=0.645905773557\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:12:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=4000 train binary_classification_cross_entropy <loss>=0.645731029294\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:12:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=4000 train binary_f_1.000 <score>=0.6443397484\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:12:16 INFO 140667034941248] Iter[0] Batch [4000]#011Speed: 22001.26 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:12:16 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=4000 train binary_classification_accuracy <score>=0.640536615846\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:12:16 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=4000 train binary_classification_cross_entropy <loss>=0.648038823607\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:12:16 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=4000 train binary_f_1.000 <score>=0.639601525478\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:12:16 INFO 140009951377216] Iter[0] Batch [4000]#011Speed: 22501.50 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:12:16 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=4000 train binary_classification_accuracy <score>=0.647910272432\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:12:16 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=4000 train binary_classification_cross_entropy <loss>=0.644877172339\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:12:16 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=4000 train binary_f_1.000 <score>=0.647001086277\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[08/29/2018 13:12:38 INFO 140658308478784] Iter[0] Batch [4500]#011Speed: 21084.36 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:12:38 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=4500 train binary_classification_accuracy <score>=0.64327438347\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:12:38 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=4500 train binary_classification_cross_entropy <loss>=0.646116332682\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:12:38 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=4500 train binary_f_1.000 <score>=0.642830004004\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:12:40 INFO 140667034941248] Iter[0] Batch [4500]#011Speed: 20414.06 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:12:40 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=4500 train binary_classification_accuracy <score>=0.644221950678\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:12:40 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=4500 train binary_classification_cross_entropy <loss>=0.64544417663\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:12:40 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=4500 train binary_f_1.000 <score>=0.643235517889\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:12:41 INFO 140009951377216] Iter[0] Batch [4500]#011Speed: 20572.60 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:12:41 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=4500 train binary_classification_accuracy <score>=0.651149966674\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:12:41 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=4500 train binary_classification_cross_entropy <loss>=0.642487874846\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:12:41 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=4500 train binary_f_1.000 <score>=0.650120461095\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:12:42 INFO 140684726773568] Iter[0] Batch [4500]#011Speed: 19403.45 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:12:42 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=4500 train binary_classification_accuracy <score>=0.64971006443\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:12:42 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=4500 train binary_classification_cross_entropy <loss>=0.643112122775\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:12:42 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=4500 train binary_f_1.000 <score>=0.648495769991\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:13:02 INFO 140658308478784] Iter[0] Batch [5000]#011Speed: 20449.89 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:13:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=5000 train binary_classification_accuracy <score>=0.647122775445\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:13:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=5000 train binary_classification_cross_entropy <loss>=0.643563336051\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:13:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=5000 train binary_f_1.000 <score>=0.646860727887\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:13:03 INFO 140667034941248] Iter[0] Batch [5000]#011Speed: 21936.53 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:13:03 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=5000 train binary_classification_accuracy <score>=0.648225154969\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:13:03 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=5000 train binary_classification_cross_entropy <loss>=0.642803890174\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:13:03 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=5000 train binary_f_1.000 <score>=0.647539968721\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:13:04 INFO 140009951377216] Iter[0] Batch [5000]#011Speed: 21412.79 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:13:04 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=5000 train binary_classification_accuracy <score>=0.654327534493\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:13:04 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=5000 train binary_classification_cross_entropy <loss>=0.640120028729\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:13:04 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=5000 train binary_f_1.000 <score>=0.653329739707\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:13:05 INFO 140684726773568] Iter[0] Batch [5000]#011Speed: 21434.15 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:13:05 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=5000 train binary_classification_accuracy <score>=0.653577284543\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:13:05 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=5000 train binary_classification_cross_entropy <loss>=0.640406328024\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:13:05 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=5000 train binary_f_1.000 <score>=0.652691825791\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:13:27 INFO 140658308478784] Iter[0] Batch [5500]#011Speed: 19894.29 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:13:27 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=5500 train binary_classification_accuracy <score>=0.650268860207\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:13:27 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=5500 train binary_classification_cross_entropy <loss>=0.641153279726\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:13:27 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=5500 train binary_f_1.000 <score>=0.650125437714\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:13:29 INFO 140009951377216] Iter[0] Batch [5500]#011Speed: 20096.99 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:13:29 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=5500 train binary_classification_accuracy <score>=0.657504817306\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:13:29 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=5500 train binary_classification_cross_entropy <loss>=0.637730824409\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:13:29 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=5500 train binary_f_1.000 <score>=0.656582252778\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:13:30 INFO 140667034941248] Iter[0] Batch [5500]#011Speed: 18718.80 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:13:30 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=5500 train binary_classification_accuracy <score>=0.651720596255\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:13:30 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=5500 train binary_classification_cross_entropy <loss>=0.640213547976\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:13:30 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=5500 train binary_f_1.000 <score>=0.651299886811\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:13:32 INFO 140684726773568] Iter[0] Batch [5500]#011Speed: 18887.98 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:13:32 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=5500 train binary_classification_accuracy <score>=0.657040174514\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:13:32 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=5500 train binary_classification_cross_entropy <loss>=0.637759375681\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:13:32 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=5500 train binary_f_1.000 <score>=0.65644574634\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:13:50 INFO 140658308478784] Iter[0] Batch [6000]#011Speed: 22303.43 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:13:50 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=6000 train binary_classification_accuracy <score>=0.653480753208\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:13:50 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=6000 train binary_classification_cross_entropy <loss>=0.638693846558\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:13:50 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=6000 train binary_f_1.000 <score>=0.653565038334\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:13:51 INFO 140009951377216] Iter[0] Batch [6000]#011Speed: 22707.63 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:13:51 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=6000 train binary_classification_accuracy <score>=0.660058990168\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:13:51 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=6000 train binary_classification_cross_entropy <loss>=0.635498392002\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:13:51 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=6000 train binary_f_1.000 <score>=0.659295712641\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:13:54 INFO 140684726773568] Iter[0] Batch [6000]#011Speed: 22724.52 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:13:54 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=6000 train binary_classification_accuracy <score>=0.660159473421\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:13:54 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=6000 train binary_classification_cross_entropy <loss>=0.6351934204\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:13:54 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=6000 train binary_f_1.000 <score>=0.659961547356\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:13:54 INFO 140667034941248] Iter[0] Batch [6000]#011Speed: 21009.51 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:13:54 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=6000 train binary_classification_accuracy <score>=0.655020496584\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:13:54 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=6000 train binary_classification_cross_entropy <loss>=0.637631409536\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:13:54 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=6000 train binary_f_1.000 <score>=0.654938960748\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[08/29/2018 13:14:12 INFO 140658308478784] Iter[0] Batch [6500]#011Speed: 22091.65 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:14:12 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=6500 train binary_classification_accuracy <score>=0.656163974773\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:14:12 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=6500 train binary_classification_cross_entropy <loss>=0.636362072696\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:14:12 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=6500 train binary_f_1.000 <score>=0.656418925566\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:14:15 INFO 140009951377216] Iter[0] Batch [6500]#011Speed: 20727.74 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:14:15 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=6500 train binary_classification_accuracy <score>=0.662223350254\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:14:15 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=6500 train binary_classification_cross_entropy <loss>=0.633362014928\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:14:15 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=6500 train binary_f_1.000 <score>=0.661723610761\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:14:18 INFO 140667034941248] Iter[0] Batch [6500]#011Speed: 20843.76 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:14:18 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=6500 train binary_classification_accuracy <score>=0.658\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:14:18 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=6500 train binary_classification_cross_entropy <loss>=0.635054286824\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:14:18 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=6500 train binary_f_1.000 <score>=0.658155226238\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:14:18 INFO 140684726773568] Iter[0] Batch [6500]#011Speed: 21065.34 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:14:18 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=6500 train binary_classification_accuracy <score>=0.662615289955\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:14:18 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=6500 train binary_classification_cross_entropy <loss>=0.632889925232\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:14:18 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=6500 train binary_f_1.000 <score>=0.662607712771\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:14:35 INFO 140658308478784] Iter[0] Batch [7000]#011Speed: 21789.82 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:14:35 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=7000 train binary_classification_accuracy <score>=0.658668904442\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:14:35 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=7000 train binary_classification_cross_entropy <loss>=0.634107905789\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:14:35 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=7000 train binary_f_1.000 <score>=0.659038405316\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:14:39 INFO 140009951377216] Iter[0] Batch [7000]#011Speed: 21057.64 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:14:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=7000 train binary_classification_accuracy <score>=0.664393800886\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:14:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=7000 train binary_classification_cross_entropy <loss>=0.631222729307\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:14:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=7000 train binary_f_1.000 <score>=0.663960515866\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:14:40 INFO 140667034941248] Iter[0] Batch [7000]#011Speed: 22439.75 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:14:40 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=7000 train binary_classification_accuracy <score>=0.660602056849\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:14:40 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=7000 train binary_classification_cross_entropy <loss>=0.632687350982\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:14:40 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=7000 train binary_f_1.000 <score>=0.660898050566\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:14:41 INFO 140684726773568] Iter[0] Batch [7000]#011Speed: 21483.79 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:14:41 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=7000 train binary_classification_accuracy <score>=0.66497328953\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:14:41 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=7000 train binary_classification_cross_entropy <loss>=0.630589303117\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:14:41 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=7000 train binary_f_1.000 <score>=0.66509642077\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:14:59 INFO 140658308478784] Iter[0] Batch [7500]#011Speed: 20747.71 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:14:59 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=7500 train binary_classification_accuracy <score>=0.660997866951\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:14:59 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=7500 train binary_classification_cross_entropy <loss>=0.63197308457\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:14:59 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=7500 train binary_f_1.000 <score>=0.661481432669\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:15:03 INFO 140009951377216] Iter[0] Batch [7500]#011Speed: 20407.32 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:15:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=7500 train binary_classification_accuracy <score>=0.666791627783\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:15:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=7500 train binary_classification_cross_entropy <loss>=0.629061127934\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:15:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=7500 train binary_f_1.000 <score>=0.666413791704\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:15:04 INFO 140667034941248] Iter[0] Batch [7500]#011Speed: 20513.87 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:15:04 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=7500 train binary_classification_accuracy <score>=0.663411278496\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:15:04 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=7500 train binary_classification_cross_entropy <loss>=0.63034782654\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:15:04 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=7500 train binary_f_1.000 <score>=0.663611918448\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:15:05 INFO 140684726773568] Iter[0] Batch [7500]#011Speed: 21001.12 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:15:05 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=7500 train binary_classification_accuracy <score>=0.66763804826\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:15:05 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=7500 train binary_classification_cross_entropy <loss>=0.628287543375\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:15:05 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=7500 train binary_f_1.000 <score>=0.667959553361\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:15:21 INFO 140658308478784] Iter[0] Batch [8000]#011Speed: 22567.86 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:15:21 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=8000 train binary_classification_accuracy <score>=0.663498062742\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:15:21 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=8000 train binary_classification_cross_entropy <loss>=0.629804074845\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:15:21 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=8000 train binary_f_1.000 <score>=0.664120118496\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:15:25 INFO 140009951377216] Iter[0] Batch [8000]#011Speed: 22888.60 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:15:25 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=8000 train binary_classification_accuracy <score>=0.669024246969\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:15:25 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=8000 train binary_classification_cross_entropy <loss>=0.626958845392\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:15:25 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=8000 train binary_f_1.000 <score>=0.668841097595\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:15:27 INFO 140667034941248] Iter[0] Batch [8000]#011Speed: 22473.97 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:15:27 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=8000 train binary_classification_accuracy <score>=0.665922134733\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:15:27 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=8000 train binary_classification_cross_entropy <loss>=0.628082635171\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:15:27 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=8000 train binary_f_1.000 <score>=0.666253628598\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:15:26 INFO 140684726773568] Iter[0] Batch [8000]#011Speed: 23078.16 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:15:26 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=8000 train binary_classification_accuracy <score>=0.669952130984\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:15:26 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=8000 train binary_classification_cross_entropy <loss>=0.62613293769\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:15:26 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=8000 train binary_f_1.000 <score>=0.670383750712\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[08/29/2018 13:15:45 INFO 140658308478784] Iter[0] Batch [8500]#011Speed: 21341.58 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:15:45 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=8500 train binary_classification_accuracy <score>=0.665488530761\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:15:45 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=8500 train binary_classification_cross_entropy <loss>=0.627822562759\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:15:45 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=8500 train binary_f_1.000 <score>=0.666290516967\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:15:48 INFO 140009951377216] Iter[0] Batch [8500]#011Speed: 21720.80 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:15:48 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=8500 train binary_classification_accuracy <score>=0.670714974709\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:15:48 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=8500 train binary_classification_cross_entropy <loss>=0.625080438255\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:15:48 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=8500 train binary_f_1.000 <score>=0.670808763813\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:15:49 INFO 140667034941248] Iter[0] Batch [8500]#011Speed: 22618.71 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:15:49 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=8500 train binary_classification_accuracy <score>=0.667940242324\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:15:49 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=8500 train binary_classification_cross_entropy <loss>=0.626022901473\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:15:49 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=8500 train binary_f_1.000 <score>=0.668512595239\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:15:50 INFO 140684726773568] Iter[0] Batch [8500]#011Speed: 20923.97 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:15:50 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=8500 train binary_classification_accuracy <score>=0.671801317492\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:15:50 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=8500 train binary_classification_cross_entropy <loss>=0.624143882415\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:15:50 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=8500 train binary_f_1.000 <score>=0.672357050193\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:16:08 INFO 140658308478784] Iter[0] Batch [9000]#011Speed: 21319.41 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:16:08 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=9000 train binary_classification_accuracy <score>=0.667419175647\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:16:08 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=9000 train binary_classification_cross_entropy <loss>=0.625870414402\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:16:08 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=9000 train binary_f_1.000 <score>=0.668336969533\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:16:13 INFO 140009951377216] Iter[0] Batch [9000]#011Speed: 20428.21 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:16:13 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=9000 train binary_classification_accuracy <score>=0.672418842351\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:16:13 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=9000 train binary_classification_cross_entropy <loss>=0.623206953113\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:16:13 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=9000 train binary_f_1.000 <score>=0.672675508972\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:16:13 INFO 140667034941248] Iter[0] Batch [9000]#011Speed: 20723.98 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:16:13 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=9000 train binary_classification_accuracy <score>=0.669822353072\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:16:13 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=9000 train binary_classification_cross_entropy <loss>=0.624050152936\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:16:13 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=9000 train binary_f_1.000 <score>=0.670478430846\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:16:15 INFO 140684726773568] Iter[0] Batch [9000]#011Speed: 20021.60 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:16:15 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=9000 train binary_classification_accuracy <score>=0.673713031885\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:16:15 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=9000 train binary_classification_cross_entropy <loss>=0.622122350467\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:16:15 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=9000 train binary_f_1.000 <score>=0.674314000529\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:16:30 INFO 140658308478784] Iter[0] Batch [9500]#011Speed: 23361.34 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:16:30 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=9500 train binary_classification_accuracy <score>=0.669159246395\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:16:30 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=9500 train binary_classification_cross_entropy <loss>=0.624028666686\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:16:30 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=9500 train binary_f_1.000 <score>=0.670156657114\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:16:37 INFO 140009951377216] Iter[0] Batch [9500]#011Speed: 20796.12 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:16:37 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=9500 train binary_classification_accuracy <score>=0.67421924008\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:16:37 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=9500 train binary_classification_cross_entropy <loss>=0.621328670867\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:16:37 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=9500 train binary_f_1.000 <score>=0.674622289262\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:16:37 INFO 140667034941248] Iter[0] Batch [9500]#011Speed: 20542.17 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:16:37 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=9500 train binary_classification_accuracy <score>=0.671827597095\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:16:37 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=9500 train binary_classification_cross_entropy <loss>=0.622065439799\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:16:37 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=9500 train binary_f_1.000 <score>=0.672638953576\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:16:38 INFO 140684726773568] Iter[0] Batch [9500]#011Speed: 21709.30 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:16:38 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=9500 train binary_classification_accuracy <score>=0.675622355542\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:16:38 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=9500 train binary_classification_cross_entropy <loss>=0.620149122782\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:16:38 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=9500 train binary_f_1.000 <score>=0.676277772946\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:16:54 INFO 140658308478784] Iter[0] Batch [10000]#011Speed: 20914.30 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:16:54 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=10000 train binary_classification_accuracy <score>=0.670932606739\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:16:54 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=10000 train binary_classification_cross_entropy <loss>=0.622216306946\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:16:54 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=10000 train binary_f_1.000 <score>=0.671989253297\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:17:01 INFO 140009951377216] Iter[0] Batch [10000]#011Speed: 20576.87 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:17:01 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=10000 train binary_classification_accuracy <score>=0.675628137186\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:17:01 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=10000 train binary_classification_cross_entropy <loss>=0.619626416124\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:17:01 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=10000 train binary_f_1.000 <score>=0.676176800731\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:17:03 INFO 140667034941248] Iter[0] Batch [10000]#011Speed: 19560.85 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:17:03 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=10000 train binary_classification_accuracy <score>=0.673451154885\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:17:03 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=10000 train binary_classification_cross_entropy <loss>=0.620260686444\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:17:03 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=10000 train binary_f_1.000 <score>=0.674347943534\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:17:03 INFO 140684726773568] Iter[0] Batch [10000]#011Speed: 20230.49 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:17:03 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=10000 train binary_classification_accuracy <score>=0.677150484952\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:17:03 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=10000 train binary_classification_cross_entropy <loss>=0.61837373391\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:17:03 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=10000 train binary_f_1.000 <score>=0.677948834366\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[08/29/2018 13:17:18 INFO 140658308478784] Iter[0] Batch [10500]#011Speed: 20239.17 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:17:18 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=10500 train binary_classification_accuracy <score>=0.672374916675\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:17:18 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=10500 train binary_classification_cross_entropy <loss>=0.620572093476\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:17:18 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=10500 train binary_f_1.000 <score>=0.673475882243\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:17:25 INFO 140009951377216] Iter[0] Batch [10500]#011Speed: 20558.99 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:17:25 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=10500 train binary_classification_accuracy <score>=0.677055804209\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:17:25 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=10500 train binary_classification_cross_entropy <loss>=0.61796598769\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:17:25 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=10500 train binary_f_1.000 <score>=0.677779871286\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:17:27 INFO 140667034941248] Iter[0] Batch [10500]#011Speed: 20491.96 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:17:27 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=10500 train binary_classification_accuracy <score>=0.675128463956\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:17:27 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=10500 train binary_classification_cross_entropy <loss>=0.61849883066\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:17:27 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=10500 train binary_f_1.000 <score>=0.676093736084\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:17:28 INFO 140684726773568] Iter[0] Batch [10500]#011Speed: 20316.82 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:17:28 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=10500 train binary_classification_accuracy <score>=0.678695838492\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:17:28 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=10500 train binary_classification_cross_entropy <loss>=0.616652763556\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:17:28 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=10500 train binary_f_1.000 <score>=0.679514402838\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:17:43 INFO 140658308478784] Iter[0] Batch [11000]#011Speed: 19992.21 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:17:43 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=11000 train binary_classification_accuracy <score>=0.67386928461\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:17:43 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=11000 train binary_classification_cross_entropy <loss>=0.618928964699\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:17:43 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=11000 train binary_f_1.000 <score>=0.67505569611\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:17:49 INFO 140009951377216] Iter[0] Batch [11000]#011Speed: 20859.13 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:17:49 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=11000 train binary_classification_accuracy <score>=0.678242614308\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:17:49 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=11000 train binary_classification_cross_entropy <loss>=0.616433835544\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:17:49 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=11000 train binary_f_1.000 <score>=0.679032967543\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:17:52 INFO 140667034941248] Iter[0] Batch [11000]#011Speed: 20257.81 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:17:52 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=11000 train binary_classification_accuracy <score>=0.676570766294\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:17:52 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=11000 train binary_classification_cross_entropy <loss>=0.616853233929\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:17:52 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=11000 train binary_f_1.000 <score>=0.677690922911\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:17:53 INFO 140684726773568] Iter[0] Batch [11000]#011Speed: 19606.11 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:17:53 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=11000 train binary_classification_accuracy <score>=0.680054722298\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:17:53 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=11000 train binary_classification_cross_entropy <loss>=0.615040559562\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:17:53 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=11000 train binary_f_1.000 <score>=0.680993733381\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:18:07 INFO 140658308478784] Iter[0] Batch [11500]#011Speed: 21572.20 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:18:07 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=11500 train binary_classification_accuracy <score>=0.675199287019\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:18:07 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=11500 train binary_classification_cross_entropy <loss>=0.617415593373\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:18:07 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=11500 train binary_f_1.000 <score>=0.676416502292\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:18:12 INFO 140009951377216] Iter[0] Batch [11500]#011Speed: 22092.16 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:18:12 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=11500 train binary_classification_accuracy <score>=0.679588383619\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:18:12 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=11500 train binary_classification_cross_entropy <loss>=0.614900855492\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:18:12 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=11500 train binary_f_1.000 <score>=0.680489269972\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:18:13 INFO 140667034941248] Iter[0] Batch [11500]#011Speed: 23168.21 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:18:13 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=11500 train binary_classification_accuracy <score>=0.678009129641\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:18:13 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=11500 train binary_classification_cross_entropy <loss>=0.61525970364\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:18:13 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=11500 train binary_f_1.000 <score>=0.679205977802\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:18:16 INFO 140684726773568] Iter[0] Batch [11500]#011Speed: 21693.00 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:18:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=11500 train binary_classification_accuracy <score>=0.681445352578\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:18:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=11500 train binary_classification_cross_entropy <loss>=0.6134419635\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:18:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=11500 train binary_f_1.000 <score>=0.682442369228\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:18:30 INFO 140658308478784] Iter[0] Batch [12000]#011Speed: 20903.07 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:18:30 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=12000 train binary_classification_accuracy <score>=0.676553537205\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:18:30 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=12000 train binary_classification_cross_entropy <loss>=0.615901929895\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:18:30 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=12000 train binary_f_1.000 <score>=0.677827649523\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:18:35 INFO 140009951377216] Iter[0] Batch [12000]#011Speed: 21636.52 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:18:35 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=12000 train binary_classification_accuracy <score>=0.68084676277\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:18:35 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=12000 train binary_classification_cross_entropy <loss>=0.613433754936\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:18:35 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=12000 train binary_f_1.000 <score>=0.681830143847\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:18:37 INFO 140667034941248] Iter[0] Batch [12000]#011Speed: 21503.97 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:18:37 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=12000 train binary_classification_accuracy <score>=0.679398383468\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:18:37 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=12000 train binary_classification_cross_entropy <loss>=0.613723037891\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:18:37 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=12000 train binary_f_1.000 <score>=0.680665238563\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:18:40 INFO 140684726773568] Iter[0] Batch [12000]#011Speed: 21179.69 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:18:40 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=12000 train binary_classification_accuracy <score>=0.682729022581\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:18:40 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=12000 train binary_classification_cross_entropy <loss>=0.611920545526\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:18:40 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=12000 train binary_f_1.000 <score>=0.683758014606\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[08/29/2018 13:18:53 INFO 140658308478784] Iter[0] Batch [12500]#011Speed: 22520.90 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:18:53 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=12500 train binary_classification_accuracy <score>=0.67776937845\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:18:53 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=12500 train binary_classification_cross_entropy <loss>=0.614453260536\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:18:53 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=12500 train binary_f_1.000 <score>=0.679070936931\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:18:58 INFO 140009951377216] Iter[0] Batch [12500]#011Speed: 21557.92 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:18:58 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=12500 train binary_classification_accuracy <score>=0.68196512279\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:18:58 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=12500 train binary_classification_cross_entropy <loss>=0.612016037106\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:18:58 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=12500 train binary_f_1.000 <score>=0.682975310892\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:19:01 INFO 140667034941248] Iter[0] Batch [12500]#011Speed: 20531.58 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:19:01 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=12500 train binary_classification_accuracy <score>=0.680695704344\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:19:01 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=12500 train binary_classification_cross_entropy <loss>=0.61221394351\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:19:01 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=12500 train binary_f_1.000 <score>=0.682036917949\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:19:03 INFO 140684726773568] Iter[0] Batch [12500]#011Speed: 21249.31 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:19:03 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=12500 train binary_classification_accuracy <score>=0.68399288057\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:19:03 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=12500 train binary_classification_cross_entropy <loss>=0.610412292715\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:19:03 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=12500 train binary_f_1.000 <score>=0.685070935083\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:19:15 INFO 140658308478784] Iter[0] Batch [13000]#011Speed: 22616.88 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:19:15 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=13000 train binary_classification_accuracy <score>=0.678965771864\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:19:15 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=13000 train binary_classification_cross_entropy <loss>=0.613024114821\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:19:15 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=13000 train binary_f_1.000 <score>=0.68028780307\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:19:21 INFO 140009951377216] Iter[0] Batch [13000]#011Speed: 22208.64 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:19:21 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=13000 train binary_classification_accuracy <score>=0.683068148604\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:19:21 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=13000 train binary_classification_cross_entropy <loss>=0.610612757778\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:19:21 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=13000 train binary_f_1.000 <score>=0.684147092031\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:19:24 INFO 140667034941248] Iter[0] Batch [13000]#011Speed: 21282.80 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:19:24 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=13000 train binary_classification_accuracy <score>=0.681812091378\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:19:24 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=13000 train binary_classification_cross_entropy <loss>=0.610772847852\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:19:24 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=13000 train binary_f_1.000 <score>=0.683195276251\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:19:27 INFO 140684726773568] Iter[0] Batch [13000]#011Speed: 20964.54 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:19:27 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=13000 train binary_classification_accuracy <score>=0.685128220906\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:19:27 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=13000 train binary_classification_cross_entropy <loss>=0.608964964154\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:19:27 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=13000 train binary_f_1.000 <score>=0.6862117209\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:19:37 INFO 140658308478784] Iter[0] Batch [13500]#011Speed: 22237.90 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:19:37 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=13500 train binary_classification_accuracy <score>=0.680064661877\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:19:37 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=13500 train binary_classification_cross_entropy <loss>=0.611696874902\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:19:37 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=13500 train binary_f_1.000 <score>=0.681482478888\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:19:44 INFO 140009951377216] Iter[0] Batch [13500]#011Speed: 21427.91 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:19:44 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=13500 train binary_classification_accuracy <score>=0.684111991704\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:19:44 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=13500 train binary_classification_cross_entropy <loss>=0.609295800318\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:19:44 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=13500 train binary_f_1.000 <score>=0.685297005469\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:19:47 INFO 140667034941248] Iter[0] Batch [13500]#011Speed: 22537.17 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:19:47 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=13500 train binary_classification_accuracy <score>=0.683007703133\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:19:47 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=13500 train binary_classification_cross_entropy <loss>=0.609374980122\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:19:47 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=13500 train binary_f_1.000 <score>=0.684475996339\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:19:51 INFO 140684726773568] Iter[0] Batch [13500]#011Speed: 20971.95 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:19:51 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=13500 train binary_classification_accuracy <score>=0.686181616177\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:19:51 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=13500 train binary_classification_cross_entropy <loss>=0.607600950367\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:19:51 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=13500 train binary_f_1.000 <score>=0.687345023385\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:02 INFO 140658308478784] Iter[0] Batch [14000]#011Speed: 20541.77 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=14000 train binary_classification_accuracy <score>=0.681055853153\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=14000 train binary_classification_cross_entropy <loss>=0.610411216098\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, batch=14000 train binary_f_1.000 <score>=0.682561244273\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, train binary_classification_accuracy <score>=0.681055853153\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, train binary_classification_cross_entropy <loss>=0.610411216098\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=0, train binary_f_1.000 <score>=0.682561244273\u001b[0m\n",
      "\u001b[32m#metrics {\"Metrics\": {\"epochs\": {\"count\": 1, \"max\": 100, \"sum\": 100.0, \"min\": 100}, \"update.time\": {\"count\": 1, \"max\": 648452.9581069946, \"sum\": 648452.9581069946, \"min\": 648452.9581069946}}, \"EndTime\": 1535548802.098881, \"Dimensions\": {\"Host\": \"algo-2\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535548153.645627}\n",
      "\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:02 INFO 140658308478784] #progress_metric: host=algo-2, completed 1 % of epochs\u001b[0m\n",
      "\u001b[32m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Total Batches Seen\": {\"count\": 1, \"max\": 14002, \"sum\": 14002.0, \"min\": 14002}, \"Total Records Seen\": {\"count\": 1, \"max\": 14001184, \"sum\": 14001184.0, \"min\": 14001184}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Reset Count\": {\"count\": 1, \"max\": 2, \"sum\": 2.0, \"min\": 2}}, \"EndTime\": 1535548802.099148, \"Dimensions\": {\"Host\": \"algo-2\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 0}, \"StartTime\": 1535548153.645889}\n",
      "\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:02 INFO 140658308478784] #throughput_metric: host=algo-2, train throughput=21590.1165385 records/second\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=0 train binary_classification_accuracy <score>=0.718\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=0 train binary_classification_cross_entropy <loss>=0.568983276367\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=0 train binary_f_1.000 <score>=0.726213592233\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31m[08/29/2018 13:20:08 INFO 140009951377216] Iter[0] Batch [14000]#011Speed: 20425.33 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:08 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=14000 train binary_classification_accuracy <score>=0.685036711663\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:08 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=14000 train binary_classification_cross_entropy <loss>=0.608023183747\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:08 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, batch=14000 train binary_f_1.000 <score>=0.686285363584\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:08 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, train binary_classification_accuracy <score>=0.685036711663\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:08 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, train binary_classification_cross_entropy <loss>=0.608023183747\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:08 INFO 140009951377216] #quality_metric: host=algo-1, epoch=0, train binary_f_1.000 <score>=0.686285363584\u001b[0m\n",
      "\u001b[31m#metrics {\"Metrics\": {\"epochs\": {\"count\": 1, \"max\": 100, \"sum\": 100.0, \"min\": 100}, \"update.time\": {\"count\": 1, \"max\": 655312.5579357147, \"sum\": 655312.5579357147, \"min\": 655312.5579357147}}, \"EndTime\": 1535548808.95606, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535548153.643148}\n",
      "\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:08 INFO 140009951377216] #progress_metric: host=algo-1, completed 1 % of epochs\u001b[0m\n",
      "\u001b[31m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Total Batches Seen\": {\"count\": 1, \"max\": 14002, \"sum\": 14002.0, \"min\": 14002}, \"Total Records Seen\": {\"count\": 1, \"max\": 14001184, \"sum\": 14001184.0, \"min\": 14001184}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Reset Count\": {\"count\": 1, \"max\": 2, \"sum\": 2.0, \"min\": 2}}, \"EndTime\": 1535548808.956245, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 0}, \"StartTime\": 1535548153.643474}\n",
      "\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:08 INFO 140009951377216] #throughput_metric: host=algo-1, train throughput=21364.1225884 records/second\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:09 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=0 train binary_classification_accuracy <score>=0.719\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:09 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=0 train binary_classification_cross_entropy <loss>=0.567292724609\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:09 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=0 train binary_f_1.000 <score>=0.727449078565\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:10 INFO 140667034941248] Iter[0] Batch [14000]#011Speed: 21077.65 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:10 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=14000 train binary_classification_accuracy <score>=0.684037640169\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:10 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=14000 train binary_classification_cross_entropy <loss>=0.608071956011\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:10 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, batch=14000 train binary_f_1.000 <score>=0.685532623601\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:10 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, train binary_classification_accuracy <score>=0.684037640169\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:10 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, train binary_classification_cross_entropy <loss>=0.608071956011\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:10 INFO 140667034941248] #quality_metric: host=algo-3, epoch=0, train binary_f_1.000 <score>=0.685532623601\u001b[0m\n",
      "\u001b[33m#metrics {\"Metrics\": {\"epochs\": {\"count\": 1, \"max\": 100, \"sum\": 100.0, \"min\": 100}, \"update.time\": {\"count\": 1, \"max\": 657173.5498905182, \"sum\": 657173.5498905182, \"min\": 657173.5498905182}}, \"EndTime\": 1535548810.818283, \"Dimensions\": {\"Host\": \"algo-3\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535548153.64437}\n",
      "\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:10 INFO 140667034941248] #progress_metric: host=algo-3, completed 1 % of epochs\u001b[0m\n",
      "\u001b[33m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Total Batches Seen\": {\"count\": 1, \"max\": 14002, \"sum\": 14002.0, \"min\": 14002}, \"Total Records Seen\": {\"count\": 1, \"max\": 14001184, \"sum\": 14001184.0, \"min\": 14001184}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Reset Count\": {\"count\": 1, \"max\": 2, \"sum\": 2.0, \"min\": 2}}, \"EndTime\": 1535548810.818485, \"Dimensions\": {\"Host\": \"algo-3\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 0}, \"StartTime\": 1535548153.644701}\n",
      "\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:10 INFO 140667034941248] #throughput_metric: host=algo-3, train throughput=21303.6224003 records/second\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:10 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=0 train binary_classification_accuracy <score>=0.722\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:10 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=0 train binary_classification_cross_entropy <loss>=0.565207641602\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:10 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=0 train binary_f_1.000 <score>=0.739212007505\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:20:14 INFO 140684726773568] Iter[0] Batch [14000]#011Speed: 21318.98 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:20:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=14000 train binary_classification_accuracy <score>=0.687227769445\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:20:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=14000 train binary_classification_cross_entropy <loss>=0.60626302518\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:20:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, batch=14000 train binary_f_1.000 <score>=0.688495413565\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:20:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, train binary_classification_accuracy <score>=0.687227769445\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:20:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, train binary_classification_cross_entropy <loss>=0.60626302518\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:20:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=0, train binary_f_1.000 <score>=0.688495413565\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"epochs\": {\"count\": 1, \"max\": 100, \"sum\": 100.0, \"min\": 100}, \"update.time\": {\"count\": 1, \"max\": 661318.1281089783, \"sum\": 661318.1281089783, \"min\": 661318.1281089783}}, \"EndTime\": 1535548814.963529, \"Dimensions\": {\"Host\": \"algo-4\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535548153.645121}\n",
      "\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:20:14 INFO 140684726773568] #progress_metric: host=algo-4, completed 1 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Total Batches Seen\": {\"count\": 1, \"max\": 14002, \"sum\": 14002.0, \"min\": 14002}, \"Total Records Seen\": {\"count\": 1, \"max\": 14001184, \"sum\": 14001184.0, \"min\": 14001184}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Reset Count\": {\"count\": 1, \"max\": 2, \"sum\": 2.0, \"min\": 2}}, \"EndTime\": 1535548814.963794, \"Dimensions\": {\"Host\": \"algo-4\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 0}, \"StartTime\": 1535548153.645372}\n",
      "\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:20:14 INFO 140684726773568] #throughput_metric: host=algo-4, train throughput=21170.1062329 records/second\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:20:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=0 train binary_classification_accuracy <score>=0.681\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:20:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=0 train binary_classification_cross_entropy <loss>=0.583447387695\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:20:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=0 train binary_f_1.000 <score>=0.61978545888\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:24 INFO 140658308478784] Iter[1] Batch [500]#011Speed: 22092.26 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:24 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=500 train binary_classification_accuracy <score>=0.713874251497\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:24 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=500 train binary_classification_cross_entropy <loss>=0.572204914755\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:24 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=500 train binary_f_1.000 <score>=0.716909143691\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:32 INFO 140009951377216] Iter[1] Batch [500]#011Speed: 21341.95 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:32 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=500 train binary_classification_accuracy <score>=0.716686626747\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:32 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=500 train binary_classification_cross_entropy <loss>=0.569778549103\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:32 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=500 train binary_f_1.000 <score>=0.720523432793\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:33 INFO 140667034941248] Iter[1] Batch [500]#011Speed: 22269.43 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:33 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=500 train binary_classification_accuracy <score>=0.718530938124\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:33 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=500 train binary_classification_cross_entropy <loss>=0.568503522252\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:33 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=500 train binary_f_1.000 <score>=0.721685172832\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m[08/29/2018 13:20:38 INFO 140684726773568] Iter[1] Batch [500]#011Speed: 20880.31 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:20:38 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=500 train binary_classification_accuracy <score>=0.718419161677\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:20:38 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=500 train binary_classification_cross_entropy <loss>=0.567465844678\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:20:38 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=500 train binary_f_1.000 <score>=0.723222168595\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:49 INFO 140658308478784] Iter[1] Batch [1000]#011Speed: 20425.32 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:49 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=1000 train binary_classification_accuracy <score>=0.715526473526\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:49 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=1000 train binary_classification_cross_entropy <loss>=0.570205087027\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:20:49 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=1000 train binary_f_1.000 <score>=0.718231005494\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:56 INFO 140009951377216] Iter[1] Batch [1000]#011Speed: 21354.67 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:56 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=1000 train binary_classification_accuracy <score>=0.716308691309\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:56 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=1000 train binary_classification_cross_entropy <loss>=0.568562020987\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:20:56 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=1000 train binary_f_1.000 <score>=0.718859737269\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:56 INFO 140667034941248] Iter[1] Batch [1000]#011Speed: 21156.00 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:56 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=1000 train binary_classification_accuracy <score>=0.717981018981\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:56 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=1000 train binary_classification_cross_entropy <loss>=0.567441483443\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:20:56 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=1000 train binary_f_1.000 <score>=0.7208254755\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:21:02 INFO 140684726773568] Iter[1] Batch [1000]#011Speed: 21587.36 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:21:02 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=1000 train binary_classification_accuracy <score>=0.720006993007\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:21:02 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=1000 train binary_classification_cross_entropy <loss>=0.565474255664\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:21:02 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=1000 train binary_f_1.000 <score>=0.723809917982\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:21:13 INFO 140658308478784] Iter[1] Batch [1500]#011Speed: 20878.63 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:21:13 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=1500 train binary_classification_accuracy <score>=0.716215189873\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:21:13 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=1500 train binary_classification_cross_entropy <loss>=0.568804952948\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:21:13 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=1500 train binary_f_1.000 <score>=0.718819620794\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:21:19 INFO 140009951377216] Iter[1] Batch [1500]#011Speed: 21673.46 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:21:19 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=1500 train binary_classification_accuracy <score>=0.717459027315\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:21:19 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=1500 train binary_classification_cross_entropy <loss>=0.567068395443\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:21:19 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=1500 train binary_f_1.000 <score>=0.720098419039\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:21:21 INFO 140667034941248] Iter[1] Batch [1500]#011Speed: 20602.23 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:21:21 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=1500 train binary_classification_accuracy <score>=0.718495669554\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:21:21 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=1500 train binary_classification_cross_entropy <loss>=0.566091291678\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:21:21 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=1500 train binary_f_1.000 <score>=0.721996915595\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:21:26 INFO 140684726773568] Iter[1] Batch [1500]#011Speed: 20101.11 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:21:26 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=1500 train binary_classification_accuracy <score>=0.722067288474\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:21:26 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=1500 train binary_classification_cross_entropy <loss>=0.563546547216\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:21:26 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=1500 train binary_f_1.000 <score>=0.725779405897\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:21:38 INFO 140658308478784] Iter[1] Batch [2000]#011Speed: 19415.24 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:21:38 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=2000 train binary_classification_accuracy <score>=0.717420289855\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:21:38 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=2000 train binary_classification_cross_entropy <loss>=0.567472412811\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:21:38 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=2000 train binary_f_1.000 <score>=0.720229736964\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:21:43 INFO 140009951377216] Iter[1] Batch [2000]#011Speed: 20450.35 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:21:43 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=2000 train binary_classification_accuracy <score>=0.718643678161\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:21:43 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=2000 train binary_classification_cross_entropy <loss>=0.565812300759\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:21:43 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=2000 train binary_f_1.000 <score>=0.720980414896\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:21:46 INFO 140667034941248] Iter[1] Batch [2000]#011Speed: 20047.11 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:21:46 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=2000 train binary_classification_accuracy <score>=0.720082958521\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:21:46 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=2000 train binary_classification_cross_entropy <loss>=0.564654284186\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:21:46 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=2000 train binary_f_1.000 <score>=0.723352404961\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:21:54 INFO 140684726773568] Iter[1] Batch [2000]#011Speed: 18272.56 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:21:54 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=2000 train binary_classification_accuracy <score>=0.72279910045\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:21:54 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=2000 train binary_classification_cross_entropy <loss>=0.562368559922\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:21:54 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=2000 train binary_f_1.000 <score>=0.72640424748\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:22:01 INFO 140658308478784] Iter[1] Batch [2500]#011Speed: 21947.61 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:22:01 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=2500 train binary_classification_accuracy <score>=0.718279488205\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:22:01 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=2500 train binary_classification_cross_entropy <loss>=0.566383127047\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:22:01 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=2500 train binary_f_1.000 <score>=0.721133603816\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:22:06 INFO 140009951377216] Iter[1] Batch [2500]#011Speed: 22314.97 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:22:06 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=2500 train binary_classification_accuracy <score>=0.719403838465\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:22:06 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=2500 train binary_classification_cross_entropy <loss>=0.564845095043\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:22:06 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=2500 train binary_f_1.000 <score>=0.721577228218\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33m[08/29/2018 13:22:07 INFO 140667034941248] Iter[1] Batch [2500]#011Speed: 22899.53 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:22:07 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=2500 train binary_classification_accuracy <score>=0.720742902839\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:22:07 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=2500 train binary_classification_cross_entropy <loss>=0.563688316885\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:22:07 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=2500 train binary_f_1.000 <score>=0.723862014431\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:22:17 INFO 140684726773568] Iter[1] Batch [2500]#011Speed: 21472.62 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:22:17 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=2500 train binary_classification_accuracy <score>=0.723477808876\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:22:17 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=2500 train binary_classification_cross_entropy <loss>=0.561252869848\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:22:17 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=2500 train binary_f_1.000 <score>=0.726906491865\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:22:26 INFO 140658308478784] Iter[1] Batch [3000]#011Speed: 20128.70 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:22:26 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=3000 train binary_classification_accuracy <score>=0.718801399533\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:22:26 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=3000 train binary_classification_cross_entropy <loss>=0.565512565168\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:22:26 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=3000 train binary_f_1.000 <score>=0.721761702961\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:22:32 INFO 140009951377216] Iter[1] Batch [3000]#011Speed: 19023.34 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:22:32 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=3000 train binary_classification_accuracy <score>=0.720099633456\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:22:32 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=3000 train binary_classification_cross_entropy <loss>=0.563970807989\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:22:32 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=3000 train binary_f_1.000 <score>=0.722379892711\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:22:33 INFO 140667034941248] Iter[1] Batch [3000]#011Speed: 19799.18 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:22:33 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=3000 train binary_classification_accuracy <score>=0.721371876041\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:22:33 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=3000 train binary_classification_cross_entropy <loss>=0.562785523722\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:22:33 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=3000 train binary_f_1.000 <score>=0.724658269064\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:22:41 INFO 140684726773568] Iter[1] Batch [3000]#011Speed: 21015.80 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:22:41 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=3000 train binary_classification_accuracy <score>=0.724092635788\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:22:41 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=3000 train binary_classification_cross_entropy <loss>=0.560316777444\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:22:41 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=3000 train binary_f_1.000 <score>=0.727356903188\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:22:50 INFO 140658308478784] Iter[1] Batch [3500]#011Speed: 20581.70 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:22:50 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=3500 train binary_classification_accuracy <score>=0.71908540417\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:22:50 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=3500 train binary_classification_cross_entropy <loss>=0.564875051394\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:22:50 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=3500 train binary_f_1.000 <score>=0.722029933699\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:22:56 INFO 140009951377216] Iter[1] Batch [3500]#011Speed: 20569.11 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:22:56 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=3500 train binary_classification_accuracy <score>=0.720655526992\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:22:56 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=3500 train binary_classification_cross_entropy <loss>=0.563236237139\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:22:56 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=3500 train binary_f_1.000 <score>=0.723030847983\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:22:56 INFO 140667034941248] Iter[1] Batch [3500]#011Speed: 21265.24 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:22:56 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=3500 train binary_classification_accuracy <score>=0.721885746929\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:22:56 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=3500 train binary_classification_cross_entropy <loss>=0.562128439499\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:22:56 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=3500 train binary_f_1.000 <score>=0.725306198679\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:23:07 INFO 140684726773568] Iter[1] Batch [3500]#011Speed: 19039.82 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:23:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=3500 train binary_classification_accuracy <score>=0.724292202228\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:23:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=3500 train binary_classification_cross_entropy <loss>=0.559791486612\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:23:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=3500 train binary_f_1.000 <score>=0.727642256924\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:23:16 INFO 140658308478784] Iter[1] Batch [4000]#011Speed: 19579.85 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:23:16 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=4000 train binary_classification_accuracy <score>=0.719670332417\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:23:16 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=4000 train binary_classification_cross_entropy <loss>=0.564084956773\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:23:16 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=4000 train binary_f_1.000 <score>=0.722693122927\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:23:19 INFO 140009951377216] Iter[1] Batch [4000]#011Speed: 21882.60 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:23:19 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=4000 train binary_classification_accuracy <score>=0.721116970757\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:23:19 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=4000 train binary_classification_cross_entropy <loss>=0.562573399298\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:23:19 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=4000 train binary_f_1.000 <score>=0.723744676047\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:23:20 INFO 140667034941248] Iter[1] Batch [4000]#011Speed: 21183.73 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:23:20 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=4000 train binary_classification_accuracy <score>=0.722312671832\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:23:20 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=4000 train binary_classification_cross_entropy <loss>=0.561488751607\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:23:20 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=4000 train binary_f_1.000 <score>=0.725727267947\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:23:29 INFO 140684726773568] Iter[1] Batch [4000]#011Speed: 23041.06 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:23:29 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=4000 train binary_classification_accuracy <score>=0.724956010997\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:23:29 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=4000 train binary_classification_cross_entropy <loss>=0.558981662299\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:23:29 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=4000 train binary_f_1.000 <score>=0.728433702175\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:23:38 INFO 140658308478784] Iter[1] Batch [4500]#011Speed: 22813.49 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:23:38 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=4500 train binary_classification_accuracy <score>=0.719766496334\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:23:38 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=4500 train binary_classification_cross_entropy <loss>=0.563553335058\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:23:38 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=4500 train binary_f_1.000 <score>=0.722778973039\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:23:41 INFO 140009951377216] Iter[1] Batch [4500]#011Speed: 23073.86 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:23:41 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=4500 train binary_classification_accuracy <score>=0.721200177738\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:23:41 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=4500 train binary_classification_cross_entropy <loss>=0.562048894774\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:23:41 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=4500 train binary_f_1.000 <score>=0.72390651952\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:23:41 INFO 140667034941248] Iter[1] Batch [4500]#011Speed: 23459.58 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:23:41 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=4500 train binary_classification_accuracy <score>=0.722373472562\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:23:41 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=4500 train binary_classification_cross_entropy <loss>=0.561034055407\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:23:41 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=4500 train binary_f_1.000 <score>=0.72569890861\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m[08/29/2018 13:23:51 INFO 140684726773568] Iter[1] Batch [4500]#011Speed: 22527.26 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:23:51 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=4500 train binary_classification_accuracy <score>=0.725345478782\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:23:51 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=4500 train binary_classification_cross_entropy <loss>=0.55831748438\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:23:51 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=4500 train binary_f_1.000 <score>=0.728822883977\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:24:00 INFO 140658308478784] Iter[1] Batch [5000]#011Speed: 22677.85 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:24:00 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=5000 train binary_classification_accuracy <score>=0.719918616277\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:24:00 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=5000 train binary_classification_cross_entropy <loss>=0.56310425\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:24:00 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=5000 train binary_f_1.000 <score>=0.722992178961\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:24:03 INFO 140009951377216] Iter[1] Batch [5000]#011Speed: 22781.40 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:24:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=5000 train binary_classification_accuracy <score>=0.721435312937\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:24:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=5000 train binary_classification_cross_entropy <loss>=0.561569085389\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:24:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=5000 train binary_f_1.000 <score>=0.724253103262\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:24:05 INFO 140667034941248] Iter[1] Batch [5000]#011Speed: 21090.25 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:24:05 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=5000 train binary_classification_accuracy <score>=0.722646470706\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:24:05 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=5000 train binary_classification_cross_entropy <loss>=0.560525105831\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:24:05 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=5000 train binary_f_1.000 <score>=0.725999944293\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:24:14 INFO 140684726773568] Iter[1] Batch [5000]#011Speed: 22189.54 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:24:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=5000 train binary_classification_accuracy <score>=0.725410117976\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:24:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=5000 train binary_classification_cross_entropy <loss>=0.557918502347\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:24:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=5000 train binary_f_1.000 <score>=0.728844001539\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:24:23 INFO 140658308478784] Iter[1] Batch [5500]#011Speed: 21393.81 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:24:23 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=5500 train binary_classification_accuracy <score>=0.720554444646\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:24:23 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=5500 train binary_classification_cross_entropy <loss>=0.562432557362\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:24:23 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=5500 train binary_f_1.000 <score>=0.723705140257\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:24:26 INFO 140009951377216] Iter[1] Batch [5500]#011Speed: 21394.33 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:24:26 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=5500 train binary_classification_accuracy <score>=0.721962006908\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:24:26 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=5500 train binary_classification_cross_entropy <loss>=0.56096479979\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:24:26 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=5500 train binary_f_1.000 <score>=0.724898506561\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:24:28 INFO 140667034941248] Iter[1] Batch [5500]#011Speed: 21777.69 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:24:28 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=5500 train binary_classification_accuracy <score>=0.723123068533\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:24:28 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=5500 train binary_classification_cross_entropy <loss>=0.5599174586\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:24:28 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=5500 train binary_f_1.000 <score>=0.72650425354\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:24:37 INFO 140684726773568] Iter[1] Batch [5500]#011Speed: 21685.41 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:24:37 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=5500 train binary_classification_accuracy <score>=0.725659698237\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:24:37 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=5500 train binary_classification_cross_entropy <loss>=0.557454742071\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:24:37 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=5500 train binary_f_1.000 <score>=0.729127516008\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:24:48 INFO 140658308478784] Iter[1] Batch [6000]#011Speed: 19938.12 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:24:48 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=6000 train binary_classification_accuracy <score>=0.720533744376\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:24:48 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=6000 train binary_classification_cross_entropy <loss>=0.562165795742\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:24:48 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=6000 train binary_f_1.000 <score>=0.72377111378\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:24:50 INFO 140009951377216] Iter[1] Batch [6000]#011Speed: 20948.51 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:24:50 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=6000 train binary_classification_accuracy <score>=0.721784202633\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:24:50 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=6000 train binary_classification_cross_entropy <loss>=0.560778526791\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:24:50 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=6000 train binary_f_1.000 <score>=0.724856266551\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:24:52 INFO 140667034941248] Iter[1] Batch [6000]#011Speed: 20384.12 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:24:52 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=6000 train binary_classification_accuracy <score>=0.722956840527\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:24:52 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=6000 train binary_classification_cross_entropy <loss>=0.559706307005\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:24:52 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=6000 train binary_f_1.000 <score>=0.726409884247\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:25:01 INFO 140684726773568] Iter[1] Batch [6000]#011Speed: 20464.67 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:25:01 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=6000 train binary_classification_accuracy <score>=0.725944509248\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:25:01 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=6000 train binary_classification_cross_entropy <loss>=0.557028324167\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:25:01 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=6000 train binary_f_1.000 <score>=0.729513035077\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:25:12 INFO 140658308478784] Iter[1] Batch [6500]#011Speed: 20980.93 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:25:12 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=6500 train binary_classification_accuracy <score>=0.720682356561\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:25:12 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=6500 train binary_classification_cross_entropy <loss>=0.561755721785\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:25:12 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=6500 train binary_f_1.000 <score>=0.724046266949\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:25:13 INFO 140009951377216] Iter[1] Batch [6500]#011Speed: 21282.41 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:25:13 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=6500 train binary_classification_accuracy <score>=0.721901553607\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:25:13 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=6500 train binary_classification_cross_entropy <loss>=0.560411337897\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:25:13 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=6500 train binary_f_1.000 <score>=0.725068865161\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:25:16 INFO 140667034941248] Iter[1] Batch [6500]#011Speed: 21177.04 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:25:16 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=6500 train binary_classification_accuracy <score>=0.723151976619\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:25:16 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=6500 train binary_classification_cross_entropy <loss>=0.559307278072\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:25:16 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=6500 train binary_f_1.000 <score>=0.726637402301\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m[08/29/2018 13:25:25 INFO 140684726773568] Iter[1] Batch [6500]#011Speed: 20850.65 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:25:25 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=6500 train binary_classification_accuracy <score>=0.725845254576\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:25:25 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=6500 train binary_classification_cross_entropy <loss>=0.556738556054\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:25:25 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=6500 train binary_f_1.000 <score>=0.72946741991\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:25:34 INFO 140658308478784] Iter[1] Batch [7000]#011Speed: 22948.28 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:25:34 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=7000 train binary_classification_accuracy <score>=0.720741465505\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:25:34 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=7000 train binary_classification_cross_entropy <loss>=0.561414761052\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:25:34 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=7000 train binary_f_1.000 <score>=0.724118365472\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:25:36 INFO 140009951377216] Iter[1] Batch [7000]#011Speed: 21711.28 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:25:36 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=7000 train binary_classification_accuracy <score>=0.722008284531\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:25:36 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=7000 train binary_classification_cross_entropy <loss>=0.560070802568\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:25:36 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=7000 train binary_f_1.000 <score>=0.725270971341\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:25:39 INFO 140667034941248] Iter[1] Batch [7000]#011Speed: 21841.07 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:25:39 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=7000 train binary_classification_accuracy <score>=0.723213540923\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:25:39 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=7000 train binary_classification_cross_entropy <loss>=0.558976868242\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:25:39 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=7000 train binary_f_1.000 <score>=0.726680918094\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:25:49 INFO 140684726773568] Iter[1] Batch [7000]#011Speed: 20666.98 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:25:49 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=7000 train binary_classification_accuracy <score>=0.725944865019\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:25:49 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=7000 train binary_classification_cross_entropy <loss>=0.556373067371\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:25:49 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=7000 train binary_f_1.000 <score>=0.729563331592\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:25:58 INFO 140658308478784] Iter[1] Batch [7500]#011Speed: 20504.66 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:25:58 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=7500 train binary_classification_accuracy <score>=0.7208697507\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:25:58 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=7500 train binary_classification_cross_entropy <loss>=0.561059969299\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:25:58 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=7500 train binary_f_1.000 <score>=0.724179450213\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:26:01 INFO 140009951377216] Iter[1] Batch [7500]#011Speed: 20472.89 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:26:01 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=7500 train binary_classification_accuracy <score>=0.722057459005\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:26:01 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=7500 train binary_classification_cross_entropy <loss>=0.559752820351\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:26:01 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=7500 train binary_f_1.000 <score>=0.725323368186\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:26:03 INFO 140667034941248] Iter[1] Batch [7500]#011Speed: 20794.98 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:26:03 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=7500 train binary_classification_accuracy <score>=0.723278496201\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:26:03 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=7500 train binary_classification_cross_entropy <loss>=0.558638339652\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:26:03 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=7500 train binary_f_1.000 <score>=0.726738545056\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:26:12 INFO 140684726773568] Iter[1] Batch [7500]#011Speed: 22305.08 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:26:12 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=7500 train binary_classification_accuracy <score>=0.725966804426\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:26:12 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=7500 train binary_classification_cross_entropy <loss>=0.556070643771\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:26:12 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=7500 train binary_f_1.000 <score>=0.72953655413\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:26:21 INFO 140658308478784] Iter[1] Batch [8000]#011Speed: 22382.30 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:26:21 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=8000 train binary_classification_accuracy <score>=0.720932008499\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:26:21 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=8000 train binary_classification_cross_entropy <loss>=0.560757780209\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:26:21 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=8000 train binary_f_1.000 <score>=0.724308701435\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:26:23 INFO 140009951377216] Iter[1] Batch [8000]#011Speed: 22771.11 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:26:23 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=8000 train binary_classification_accuracy <score>=0.722077115361\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:26:23 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=8000 train binary_classification_cross_entropy <loss>=0.55946607614\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:26:23 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=8000 train binary_f_1.000 <score>=0.725412219216\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:26:25 INFO 140667034941248] Iter[1] Batch [8000]#011Speed: 22543.39 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:26:25 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=8000 train binary_classification_accuracy <score>=0.723341207349\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:26:25 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=8000 train binary_classification_cross_entropy <loss>=0.558319743823\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:26:25 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=8000 train binary_f_1.000 <score>=0.726787994824\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:26:35 INFO 140684726773568] Iter[1] Batch [8000]#011Speed: 21020.97 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:26:35 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=8000 train binary_classification_accuracy <score>=0.725990251219\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:26:35 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=8000 train binary_classification_cross_entropy <loss>=0.555797200712\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:26:35 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=8000 train binary_f_1.000 <score>=0.729586704117\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:26:42 INFO 140658308478784] Iter[1] Batch [8500]#011Speed: 23354.03 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:26:42 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=8500 train binary_classification_accuracy <score>=0.720992236208\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:26:42 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=8500 train binary_classification_cross_entropy <loss>=0.560485799393\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:26:42 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=8500 train binary_f_1.000 <score>=0.724467587685\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:26:44 INFO 140009951377216] Iter[1] Batch [8500]#011Speed: 23107.29 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:26:44 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=8500 train binary_classification_accuracy <score>=0.72213151394\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:26:44 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=8500 train binary_classification_cross_entropy <loss>=0.559208600272\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:26:44 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=8500 train binary_f_1.000 <score>=0.725568158583\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:26:48 INFO 140667034941248] Iter[1] Batch [8500]#011Speed: 21631.56 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:26:48 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=8500 train binary_classification_accuracy <score>=0.723303376073\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:26:48 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=8500 train binary_classification_cross_entropy <loss>=0.55809519688\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:26:48 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=8500 train binary_f_1.000 <score>=0.726790469414\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m[08/29/2018 13:26:58 INFO 140684726773568] Iter[1] Batch [8500]#011Speed: 22071.08 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:26:58 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=8500 train binary_classification_accuracy <score>=0.725916480414\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:26:58 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=8500 train binary_classification_cross_entropy <loss>=0.555590243368\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:26:58 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=8500 train binary_f_1.000 <score>=0.729552018995\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:27:05 INFO 140658308478784] Iter[1] Batch [9000]#011Speed: 21524.81 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:27:05 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=9000 train binary_classification_accuracy <score>=0.720971114321\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:27:05 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=9000 train binary_classification_cross_entropy <loss>=0.560241568082\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:27:05 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=9000 train binary_f_1.000 <score>=0.724488485136\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:27:07 INFO 140009951377216] Iter[1] Batch [9000]#011Speed: 21760.88 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:27:07 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=9000 train binary_classification_accuracy <score>=0.722104099544\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:27:07 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=9000 train binary_classification_cross_entropy <loss>=0.558976200222\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:27:07 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=9000 train binary_f_1.000 <score>=0.725583647673\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:27:13 INFO 140667034941248] Iter[1] Batch [9000]#011Speed: 19978.90 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:27:13 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=9000 train binary_classification_accuracy <score>=0.723486945895\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:27:13 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=9000 train binary_classification_cross_entropy <loss>=0.557757871765\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:27:13 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=9000 train binary_f_1.000 <score>=0.727033299188\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:27:21 INFO 140684726773568] Iter[1] Batch [9000]#011Speed: 21785.17 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:27:21 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=9000 train binary_classification_accuracy <score>=0.725999222309\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:27:21 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=9000 train binary_classification_cross_entropy <loss>=0.555324770819\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:27:21 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=9000 train binary_f_1.000 <score>=0.72962292396\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:27:28 INFO 140658308478784] Iter[1] Batch [9500]#011Speed: 21711.66 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:27:28 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=9500 train binary_classification_accuracy <score>=0.72108504368\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:27:28 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=9500 train binary_classification_cross_entropy <loss>=0.559949614936\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:27:28 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=9500 train binary_f_1.000 <score>=0.724652104517\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:27:30 INFO 140009951377216] Iter[1] Batch [9500]#011Speed: 21642.34 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:27:30 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=9500 train binary_classification_accuracy <score>=0.722183138617\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:27:30 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=9500 train binary_classification_cross_entropy <loss>=0.558710537119\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:27:30 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=9500 train binary_f_1.000 <score>=0.725696452863\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:27:38 INFO 140667034941248] Iter[1] Batch [9500]#011Speed: 20049.97 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:27:38 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=9500 train binary_classification_accuracy <score>=0.723485527839\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:27:38 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=9500 train binary_classification_cross_entropy <loss>=0.557496452817\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:27:38 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=9500 train binary_f_1.000 <score>=0.727051639396\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:27:45 INFO 140684726773568] Iter[1] Batch [9500]#011Speed: 21076.04 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:27:45 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=9500 train binary_classification_accuracy <score>=0.725982843911\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:27:45 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=9500 train binary_classification_cross_entropy <loss>=0.555071611718\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:27:45 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=9500 train binary_f_1.000 <score>=0.729613973714\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:27:52 INFO 140009951377216] Iter[1] Batch [10000]#011Speed: 23656.93 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:27:52 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=10000 train binary_classification_accuracy <score>=0.722052794721\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:27:52 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=10000 train binary_classification_cross_entropy <loss>=0.558559761657\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:27:52 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=10000 train binary_f_1.000 <score>=0.725586557192\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:27:52 INFO 140658308478784] Iter[1] Batch [10000]#011Speed: 21478.23 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:27:52 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=10000 train binary_classification_accuracy <score>=0.72102619738\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:27:52 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=10000 train binary_classification_cross_entropy <loss>=0.559759530175\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:27:52 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=10000 train binary_f_1.000 <score>=0.724612784739\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:28:01 INFO 140667034941248] Iter[1] Batch [10000]#011Speed: 21960.87 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:28:01 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=10000 train binary_classification_accuracy <score>=0.723532146785\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:28:01 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=10000 train binary_classification_cross_entropy <loss>=0.557242579026\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:28:01 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=10000 train binary_f_1.000 <score>=0.727143505214\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:28:07 INFO 140684726773568] Iter[1] Batch [10000]#011Speed: 22190.59 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:28:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=10000 train binary_classification_accuracy <score>=0.72596210379\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:28:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=10000 train binary_classification_cross_entropy <loss>=0.554878626609\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:28:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=10000 train binary_f_1.000 <score>=0.729628221824\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:28:15 INFO 140658308478784] Iter[1] Batch [10500]#011Speed: 21825.97 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:28:15 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=10500 train binary_classification_accuracy <score>=0.721010379964\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:28:15 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=10500 train binary_classification_cross_entropy <loss>=0.559556591802\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:28:15 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=10500 train binary_f_1.000 <score>=0.724631472171\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:28:14 INFO 140009951377216] Iter[1] Batch [10500]#011Speed: 22014.06 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:28:14 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=10500 train binary_classification_accuracy <score>=0.721988191601\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:28:14 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=10500 train binary_classification_cross_entropy <loss>=0.558437708453\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:28:14 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=10500 train binary_f_1.000 <score>=0.725538393386\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:28:26 INFO 140667034941248] Iter[1] Batch [10500]#011Speed: 19648.92 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:28:26 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=10500 train binary_classification_accuracy <score>=0.723534615751\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:28:26 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=10500 train binary_classification_cross_entropy <loss>=0.55703676077\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:28:26 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=10500 train binary_f_1.000 <score>=0.727162925337\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:28:33 INFO 140684726773568] Iter[1] Batch [10500]#011Speed: 19682.83 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:28:33 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=10500 train binary_classification_accuracy <score>=0.725835444243\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:28:33 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=10500 train binary_classification_cross_entropy <loss>=0.554758736705\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:28:33 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=10500 train binary_f_1.000 <score>=0.729557511683\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:28:39 INFO 140009951377216] Iter[1] Batch [11000]#011Speed: 20605.46 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:28:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=11000 train binary_classification_accuracy <score>=0.721939096446\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:28:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=11000 train binary_classification_cross_entropy <loss>=0.558276187569\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:28:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=11000 train binary_f_1.000 <score>=0.72551249904\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:28:39 INFO 140658308478784] Iter[1] Batch [11000]#011Speed: 20767.29 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:28:39 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=11000 train binary_classification_accuracy <score>=0.72106263067\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:28:39 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=11000 train binary_classification_cross_entropy <loss>=0.559312669451\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:28:39 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=11000 train binary_f_1.000 <score>=0.724693147505\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33m[08/29/2018 13:28:52 INFO 140667034941248] Iter[1] Batch [11000]#011Speed: 19763.92 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:28:52 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=11000 train binary_classification_accuracy <score>=0.723462321607\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:28:52 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=11000 train binary_classification_cross_entropy <loss>=0.556843264759\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:28:52 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=11000 train binary_f_1.000 <score>=0.727125771604\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:28:56 INFO 140684726773568] Iter[1] Batch [11000]#011Speed: 21415.14 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:28:56 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=11000 train binary_classification_accuracy <score>=0.725869466412\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:28:56 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=11000 train binary_classification_cross_entropy <loss>=0.55453612239\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:28:56 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=11000 train binary_f_1.000 <score>=0.729605099407\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:29:01 INFO 140658308478784] Iter[1] Batch [11500]#011Speed: 21977.17 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:29:01 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=11500 train binary_classification_accuracy <score>=0.721022085036\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:29:01 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=11500 train binary_classification_cross_entropy <loss>=0.559145372905\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:29:01 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=11500 train binary_f_1.000 <score>=0.72469630382\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:29:03 INFO 140009951377216] Iter[1] Batch [11500]#011Speed: 20811.97 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:29:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=11500 train binary_classification_accuracy <score>=0.721896791583\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:29:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=11500 train binary_classification_cross_entropy <loss>=0.558127158681\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:29:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=11500 train binary_f_1.000 <score>=0.725466208284\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:29:14 INFO 140667034941248] Iter[1] Batch [11500]#011Speed: 22103.53 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:29:14 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=11500 train binary_classification_accuracy <score>=0.72350795583\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:29:14 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=11500 train binary_classification_cross_entropy <loss>=0.556615923333\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:29:14 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=11500 train binary_f_1.000 <score>=0.727175887552\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:29:19 INFO 140684726773568] Iter[1] Batch [11500]#011Speed: 22268.92 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:29:19 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=11500 train binary_classification_accuracy <score>=0.725781845057\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:29:19 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=11500 train binary_classification_cross_entropy <loss>=0.554388455036\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:29:19 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=11500 train binary_f_1.000 <score>=0.729550942208\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:29:24 INFO 140658308478784] Iter[1] Batch [12000]#011Speed: 22071.58 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:29:24 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=12000 train binary_classification_accuracy <score>=0.721059161737\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:29:24 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=12000 train binary_classification_cross_entropy <loss>=0.558936274287\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:29:24 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=12000 train binary_f_1.000 <score>=0.724703349816\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:29:24 INFO 140009951377216] Iter[1] Batch [12000]#011Speed: 23234.44 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:29:24 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=12000 train binary_classification_accuracy <score>=0.721912257312\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:29:24 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=12000 train binary_classification_cross_entropy <loss>=0.557947813662\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:29:24 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=12000 train binary_f_1.000 <score>=0.725519341647\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:29:39 INFO 140667034941248] Iter[1] Batch [12000]#011Speed: 20588.45 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:29:39 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=12000 train binary_classification_accuracy <score>=0.723568535955\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:29:39 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=12000 train binary_classification_cross_entropy <loss>=0.556388891888\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:29:39 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=12000 train binary_f_1.000 <score>=0.727242276348\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:29:43 INFO 140684726773568] Iter[1] Batch [12000]#011Speed: 20117.10 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:29:43 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=12000 train binary_classification_accuracy <score>=0.725792267311\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:29:43 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=12000 train binary_classification_cross_entropy <loss>=0.554200399532\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:29:43 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=12000 train binary_f_1.000 <score>=0.729569750428\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:29:49 INFO 140658308478784] Iter[1] Batch [12500]#011Speed: 19752.58 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:29:49 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=12500 train binary_classification_accuracy <score>=0.721067434605\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:29:49 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=12500 train binary_classification_cross_entropy <loss>=0.558729955056\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:29:49 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=12500 train binary_f_1.000 <score>=0.724702741402\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:29:49 INFO 140009951377216] Iter[1] Batch [12500]#011Speed: 19776.08 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:29:49 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=12500 train binary_classification_accuracy <score>=0.721906167507\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:29:49 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=12500 train binary_classification_cross_entropy <loss>=0.557759999491\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:29:49 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=12500 train binary_f_1.000 <score>=0.725510844034\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:30:03 INFO 140667034941248] Iter[1] Batch [12500]#011Speed: 20152.87 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:30:03 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=12500 train binary_classification_accuracy <score>=0.723514918806\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:30:03 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=12500 train binary_classification_cross_entropy <loss>=0.556196165152\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:30:03 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=12500 train binary_f_1.000 <score>=0.727179069115\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:30:08 INFO 140684726773568] Iter[1] Batch [12500]#011Speed: 20620.55 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:30:08 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=12500 train binary_classification_accuracy <score>=0.725771698264\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:30:08 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=12500 train binary_classification_cross_entropy <loss>=0.554010972743\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:30:08 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=12500 train binary_f_1.000 <score>=0.729589758492\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:30:14 INFO 140009951377216] Iter[1] Batch [13000]#011Speed: 20295.11 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:30:14 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=13000 train binary_classification_accuracy <score>=0.721957080225\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:30:14 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=13000 train binary_classification_cross_entropy <loss>=0.557522021262\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:30:14 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=13000 train binary_f_1.000 <score>=0.725553196733\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:30:14 INFO 140658308478784] Iter[1] Batch [13000]#011Speed: 19876.02 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:30:14 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=13000 train binary_classification_accuracy <score>=0.721173525113\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:30:14 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=13000 train binary_classification_cross_entropy <loss>=0.558432848521\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:30:14 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=13000 train binary_f_1.000 <score>=0.724766297699\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33m[08/29/2018 13:30:27 INFO 140667034941248] Iter[1] Batch [13000]#011Speed: 21522.95 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:30:27 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=13000 train binary_classification_accuracy <score>=0.723575494193\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:30:27 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=13000 train binary_classification_cross_entropy <loss>=0.555927023675\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:30:27 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=13000 train binary_f_1.000 <score>=0.727243141394\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:30:31 INFO 140684726773568] Iter[1] Batch [13000]#011Speed: 21709.78 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:30:31 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=13000 train binary_classification_accuracy <score>=0.725808630105\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:30:31 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=13000 train binary_classification_cross_entropy <loss>=0.553772125648\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:30:31 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=13000 train binary_f_1.000 <score>=0.729594227722\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:30:37 INFO 140658308478784] Iter[1] Batch [13500]#011Speed: 22012.02 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:30:37 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=13500 train binary_classification_accuracy <score>=0.721127397971\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:30:37 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=13500 train binary_classification_cross_entropy <loss>=0.558277098561\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:30:37 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=13500 train binary_f_1.000 <score>=0.724778662282\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:30:37 INFO 140009951377216] Iter[1] Batch [13500]#011Speed: 21554.46 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:30:37 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=13500 train binary_classification_accuracy <score>=0.721896822458\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:30:37 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=13500 train binary_classification_cross_entropy <loss>=0.557387474072\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:30:37 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=13500 train binary_f_1.000 <score>=0.725534722388\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:30:50 INFO 140667034941248] Iter[1] Batch [13500]#011Speed: 21249.49 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:30:50 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=13500 train binary_classification_accuracy <score>=0.723593437523\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:30:50 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=13500 train binary_classification_cross_entropy <loss>=0.555731339571\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:30:50 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=13500 train binary_f_1.000 <score>=0.727290398841\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:30:54 INFO 140684726773568] Iter[1] Batch [13500]#011Speed: 21325.38 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:30:54 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=13500 train binary_classification_accuracy <score>=0.725817198726\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:30:54 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=13500 train binary_classification_cross_entropy <loss>=0.553592051515\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:30:54 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=13500 train binary_f_1.000 <score>=0.729609712407\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:02 INFO 140658308478784] Iter[1] Batch [14000]#011Speed: 20387.53 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=14000 train binary_classification_accuracy <score>=0.721149632169\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=14000 train binary_classification_cross_entropy <loss>=0.558031164524\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, batch=14000 train binary_f_1.000 <score>=0.724843152484\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, train binary_classification_accuracy <score>=0.721149632169\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, train binary_classification_cross_entropy <loss>=0.558031164524\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=1, train binary_f_1.000 <score>=0.724843152484\u001b[0m\n",
      "\u001b[32m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 660097.2170829773, \"sum\": 660097.2170829773, \"min\": 660097.2170829773}}, \"EndTime\": 1535549462.203792, \"Dimensions\": {\"Host\": \"algo-2\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535548802.099008}\n",
      "\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:02 INFO 140658308478784] #progress_metric: host=algo-2, completed 2 % of epochs\u001b[0m\n",
      "\u001b[32m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Total Batches Seen\": {\"count\": 1, \"max\": 28003, \"sum\": 28003.0, \"min\": 28003}, \"Total Records Seen\": {\"count\": 1, \"max\": 28001368, \"sum\": 28001368.0, \"min\": 28001368}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Reset Count\": {\"count\": 1, \"max\": 3, \"sum\": 3.0, \"min\": 3}}, \"EndTime\": 1535549462.204058, \"Dimensions\": {\"Host\": \"algo-2\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 1}, \"StartTime\": 1535548802.10654}\n",
      "\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:02 INFO 140658308478784] #throughput_metric: host=algo-2, train throughput=21209.2605132 records/second\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=0 train binary_classification_accuracy <score>=0.685\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=0 train binary_classification_cross_entropy <loss>=0.579901916504\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:02 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=0 train binary_f_1.000 <score>=0.620938628159\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:01 INFO 140009951377216] Iter[1] Batch [14000]#011Speed: 20797.00 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:01 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=14000 train binary_classification_accuracy <score>=0.72193350475\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:01 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=14000 train binary_classification_cross_entropy <loss>=0.557175896182\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:01 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, batch=14000 train binary_f_1.000 <score>=0.725622308126\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:01 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, train binary_classification_accuracy <score>=0.72193350475\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:01 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, train binary_classification_cross_entropy <loss>=0.557175896182\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:01 INFO 140009951377216] #quality_metric: host=algo-1, epoch=1, train binary_f_1.000 <score>=0.725622308126\u001b[0m\n",
      "\u001b[31m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 652811.695098877, \"sum\": 652811.695098877, \"min\": 652811.695098877}}, \"EndTime\": 1535549461.771631, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535548808.956132}\n",
      "\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:01 INFO 140009951377216] #progress_metric: host=algo-1, completed 2 % of epochs\u001b[0m\n",
      "\u001b[31m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Total Batches Seen\": {\"count\": 1, \"max\": 28003, \"sum\": 28003.0, \"min\": 28003}, \"Total Records Seen\": {\"count\": 1, \"max\": 28001368, \"sum\": 28001368.0, \"min\": 28001368}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Reset Count\": {\"count\": 1, \"max\": 3, \"sum\": 3.0, \"min\": 3}}, \"EndTime\": 1535549461.772052, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 1}, \"StartTime\": 1535548808.95985}\n",
      "\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:01 INFO 140009951377216] #throughput_metric: host=algo-1, train throughput=21445.953847 records/second\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:01 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=0 train binary_classification_accuracy <score>=0.719\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:01 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=0 train binary_classification_cross_entropy <loss>=0.552539550781\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:01 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=0 train binary_f_1.000 <score>=0.711202466598\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33m[08/29/2018 13:31:12 INFO 140667034941248] Iter[1] Batch [14000]#011Speed: 22691.07 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:31:12 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=14000 train binary_classification_accuracy <score>=0.723661238483\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:31:12 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=14000 train binary_classification_cross_entropy <loss>=0.555493994727\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:31:12 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, batch=14000 train binary_f_1.000 <score>=0.727378575021\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:31:12 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, train binary_classification_accuracy <score>=0.723661238483\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:31:12 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, train binary_classification_cross_entropy <loss>=0.555493994727\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:31:12 INFO 140667034941248] #quality_metric: host=algo-3, epoch=1, train binary_f_1.000 <score>=0.727378575021\u001b[0m\n",
      "\u001b[33m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 661870.3639507294, \"sum\": 661870.3639507294, \"min\": 661870.3639507294}}, \"EndTime\": 1535549472.696221, \"Dimensions\": {\"Host\": \"algo-3\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535548810.818372}\n",
      "\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:31:12 INFO 140667034941248] #progress_metric: host=algo-3, completed 2 % of epochs\u001b[0m\n",
      "\u001b[33m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Total Batches Seen\": {\"count\": 1, \"max\": 28003, \"sum\": 28003.0, \"min\": 28003}, \"Total Records Seen\": {\"count\": 1, \"max\": 28001368, \"sum\": 28001368.0, \"min\": 28001368}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Reset Count\": {\"count\": 1, \"max\": 3, \"sum\": 3.0, \"min\": 3}}, \"EndTime\": 1535549472.696449, \"Dimensions\": {\"Host\": \"algo-3\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 1}, \"StartTime\": 1535548810.825824}\n",
      "\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:31:12 INFO 140667034941248] #throughput_metric: host=algo-3, train throughput=21152.4440399 records/second\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:31:12 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=0 train binary_classification_accuracy <score>=0.723\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:31:12 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=0 train binary_classification_cross_entropy <loss>=0.544222290039\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:31:12 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=0 train binary_f_1.000 <score>=0.755516328332\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:31:16 INFO 140684726773568] Iter[1] Batch [14000]#011Speed: 22722.80 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:31:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=14000 train binary_classification_accuracy <score>=0.725832583387\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:31:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=14000 train binary_classification_cross_entropy <loss>=0.553410979793\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:31:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, batch=14000 train binary_f_1.000 <score>=0.729633472157\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:31:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, train binary_classification_accuracy <score>=0.725832583387\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:31:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, train binary_classification_cross_entropy <loss>=0.553410979793\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:31:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=1, train binary_f_1.000 <score>=0.729633472157\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 661655.2798748016, \"sum\": 661655.2798748016, \"min\": 661655.2798748016}}, \"EndTime\": 1535549476.628334, \"Dimensions\": {\"Host\": \"algo-4\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535548814.963651}\n",
      "\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:31:16 INFO 140684726773568] #progress_metric: host=algo-4, completed 2 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Total Batches Seen\": {\"count\": 1, \"max\": 28003, \"sum\": 28003.0, \"min\": 28003}, \"Total Records Seen\": {\"count\": 1, \"max\": 28001368, \"sum\": 28001368.0, \"min\": 28001368}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Reset Count\": {\"count\": 1, \"max\": 3, \"sum\": 3.0, \"min\": 3}}, \"EndTime\": 1535549476.628863, \"Dimensions\": {\"Host\": \"algo-4\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 1}, \"StartTime\": 1535548814.972925}\n",
      "\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:31:16 INFO 140684726773568] #throughput_metric: host=algo-4, train throughput=21159.3008331 records/second\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:31:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=0 train binary_classification_accuracy <score>=0.726\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:31:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=0 train binary_classification_cross_entropy <loss>=0.543106628418\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:31:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=0 train binary_f_1.000 <score>=0.73500967118\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:25 INFO 140009951377216] Iter[2] Batch [500]#011Speed: 20701.17 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:25 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=500 train binary_classification_accuracy <score>=0.725203592814\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:25 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=500 train binary_classification_cross_entropy <loss>=0.550384624672\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:25 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=500 train binary_f_1.000 <score>=0.730076757933\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:26 INFO 140658308478784] Iter[2] Batch [500]#011Speed: 20895.33 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:26 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=500 train binary_classification_accuracy <score>=0.726181636727\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:26 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=500 train binary_classification_cross_entropy <loss>=0.549646571725\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:26 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=500 train binary_f_1.000 <score>=0.730216541821\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:31:36 INFO 140667034941248] Iter[2] Batch [500]#011Speed: 21021.07 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:31:36 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=500 train binary_classification_accuracy <score>=0.726644710579\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:31:36 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=500 train binary_classification_cross_entropy <loss>=0.547810527626\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:31:36 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=500 train binary_f_1.000 <score>=0.730993382427\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:31:41 INFO 140684726773568] Iter[2] Batch [500]#011Speed: 20332.40 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:31:41 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=500 train binary_classification_accuracy <score>=0.72879241517\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:31:41 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=500 train binary_classification_cross_entropy <loss>=0.545831345488\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:31:41 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=500 train binary_f_1.000 <score>=0.734171073509\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:48 INFO 140009951377216] Iter[2] Batch [1000]#011Speed: 22219.26 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:48 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=1000 train binary_classification_accuracy <score>=0.72508991009\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:48 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=1000 train binary_classification_cross_entropy <loss>=0.549757896035\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:31:48 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=1000 train binary_f_1.000 <score>=0.729348680257\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:50 INFO 140658308478784] Iter[2] Batch [1000]#011Speed: 20579.40 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:50 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=1000 train binary_classification_accuracy <score>=0.725895104895\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:50 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=1000 train binary_classification_cross_entropy <loss>=0.549136211994\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:31:50 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=1000 train binary_f_1.000 <score>=0.729921460583\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33m[08/29/2018 13:32:00 INFO 140667034941248] Iter[2] Batch [1000]#011Speed: 20548.51 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:32:00 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=1000 train binary_classification_accuracy <score>=0.727102897103\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:32:00 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=1000 train binary_classification_cross_entropy <loss>=0.547028568782\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:32:00 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=1000 train binary_f_1.000 <score>=0.731208611715\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:32:06 INFO 140684726773568] Iter[2] Batch [1000]#011Speed: 19860.66 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:32:06 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=1000 train binary_classification_accuracy <score>=0.72886013986\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:32:06 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=1000 train binary_classification_cross_entropy <loss>=0.545373446073\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:32:06 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=1000 train binary_f_1.000 <score>=0.733364180885\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:32:13 INFO 140009951377216] Iter[2] Batch [1500]#011Speed: 20119.86 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:32:13 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=1500 train binary_classification_accuracy <score>=0.726157228514\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:32:13 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=1500 train binary_classification_cross_entropy <loss>=0.548610596476\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:32:13 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=1500 train binary_f_1.000 <score>=0.730262415313\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:32:16 INFO 140658308478784] Iter[2] Batch [1500]#011Speed: 18952.75 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:32:16 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=1500 train binary_classification_accuracy <score>=0.726975349767\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:32:16 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=1500 train binary_classification_cross_entropy <loss>=0.547892008477\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:32:16 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=1500 train binary_f_1.000 <score>=0.731237941089\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:32:27 INFO 140667034941248] Iter[2] Batch [1500]#011Speed: 18544.07 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:32:27 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=1500 train binary_classification_accuracy <score>=0.728295136576\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:32:27 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=1500 train binary_classification_cross_entropy <loss>=0.545788261927\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:32:27 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=1500 train binary_f_1.000 <score>=0.732528176658\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:32:30 INFO 140684726773568] Iter[2] Batch [1500]#011Speed: 20576.52 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:32:30 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=1500 train binary_classification_accuracy <score>=0.729628914057\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:32:30 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=1500 train binary_classification_cross_entropy <loss>=0.544562765773\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:32:30 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=1500 train binary_f_1.000 <score>=0.733823717497\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:32:35 INFO 140009951377216] Iter[2] Batch [2000]#011Speed: 22857.13 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:32:35 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=2000 train binary_classification_accuracy <score>=0.726665167416\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:32:35 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=2000 train binary_classification_cross_entropy <loss>=0.547938258867\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:32:35 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=2000 train binary_f_1.000 <score>=0.730796320532\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:32:40 INFO 140658308478784] Iter[2] Batch [2000]#011Speed: 21572.44 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:32:40 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=2000 train binary_classification_accuracy <score>=0.727376311844\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:32:40 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=2000 train binary_classification_cross_entropy <loss>=0.547172258902\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:32:40 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=2000 train binary_f_1.000 <score>=0.731592231959\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:32:51 INFO 140667034941248] Iter[2] Batch [2000]#011Speed: 21567.19 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:32:51 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=2000 train binary_classification_accuracy <score>=0.728655172414\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:32:51 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=2000 train binary_classification_cross_entropy <loss>=0.545222369848\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:32:51 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=2000 train binary_f_1.000 <score>=0.732730202032\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:32:55 INFO 140684726773568] Iter[2] Batch [2000]#011Speed: 20467.98 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:32:55 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=2000 train binary_classification_accuracy <score>=0.729997001499\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:32:55 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=2000 train binary_classification_cross_entropy <loss>=0.543997601831\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:32:55 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=2000 train binary_f_1.000 <score>=0.733911861468\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:32:57 INFO 140009951377216] Iter[2] Batch [2500]#011Speed: 22499.49 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:32:57 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=2500 train binary_classification_accuracy <score>=0.726666133547\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:32:57 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=2500 train binary_classification_cross_entropy <loss>=0.547548306337\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:32:57 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=2500 train binary_f_1.000 <score>=0.730820177698\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:33:04 INFO 140658308478784] Iter[2] Batch [2500]#011Speed: 20522.44 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:33:04 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=2500 train binary_classification_accuracy <score>=0.727425829668\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:33:04 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=2500 train binary_classification_cross_entropy <loss>=0.546802183985\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:33:04 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=2500 train binary_f_1.000 <score>=0.731633000132\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:33:15 INFO 140667034941248] Iter[2] Batch [2500]#011Speed: 20356.22 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:33:15 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=2500 train binary_classification_accuracy <score>=0.728906837265\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:33:15 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=2500 train binary_classification_cross_entropy <loss>=0.544809442031\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:33:15 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=2500 train binary_f_1.000 <score>=0.732743111672\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:33:18 INFO 140684726773568] Iter[2] Batch [2500]#011Speed: 21319.07 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:33:18 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=2500 train binary_classification_accuracy <score>=0.730466213515\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:33:18 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=2500 train binary_classification_cross_entropy <loss>=0.543440204533\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:33:18 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=2500 train binary_f_1.000 <score>=0.734492040323\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:33:21 INFO 140009951377216] Iter[2] Batch [3000]#011Speed: 20773.02 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:33:21 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=3000 train binary_classification_accuracy <score>=0.726863712096\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:33:21 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=3000 train binary_classification_cross_entropy <loss>=0.547157872608\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:33:21 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=3000 train binary_f_1.000 <score>=0.731132443714\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:33:28 INFO 140658308478784] Iter[2] Batch [3000]#011Speed: 20442.05 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:33:28 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=3000 train binary_classification_accuracy <score>=0.727830723092\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:33:28 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=3000 train binary_classification_cross_entropy <loss>=0.546221816401\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:33:28 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=3000 train binary_f_1.000 <score>=0.732081705272\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:33:41 INFO 140667034941248] Iter[2] Batch [3000]#011Speed: 19575.12 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:33:41 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=3000 train binary_classification_accuracy <score>=0.729281572809\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:33:41 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=3000 train binary_classification_cross_entropy <loss>=0.544367993506\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:33:41 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=3000 train binary_f_1.000 <score>=0.733090349508\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:33:43 INFO 140684726773568] Iter[2] Batch [3000]#011Speed: 20457.72 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:33:43 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=3000 train binary_classification_accuracy <score>=0.730630123292\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:33:43 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=3000 train binary_classification_cross_entropy <loss>=0.543048884238\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:33:43 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=3000 train binary_f_1.000 <score>=0.734683080159\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:33:45 INFO 140009951377216] Iter[2] Batch [3500]#011Speed: 20827.88 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:33:45 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=3500 train binary_classification_accuracy <score>=0.726995144245\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:33:45 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=3500 train binary_classification_cross_entropy <loss>=0.546946802849\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:33:45 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=3500 train binary_f_1.000 <score>=0.731228241858\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[08/29/2018 13:33:52 INFO 140658308478784] Iter[2] Batch [3500]#011Speed: 21001.74 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:33:52 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=3500 train binary_classification_accuracy <score>=0.72787089403\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:33:52 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=3500 train binary_classification_cross_entropy <loss>=0.546002448938\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:33:52 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=3500 train binary_f_1.000 <score>=0.7321235569\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:34:04 INFO 140667034941248] Iter[2] Batch [3500]#011Speed: 20965.96 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:34:04 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=3500 train binary_classification_accuracy <score>=0.729427877749\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:34:04 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=3500 train binary_classification_cross_entropy <loss>=0.544145396673\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:34:04 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=3500 train binary_f_1.000 <score>=0.733332676475\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:34:07 INFO 140684726773568] Iter[2] Batch [3500]#011Speed: 20084.98 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:34:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=3500 train binary_classification_accuracy <score>=0.730712082262\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:34:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=3500 train binary_classification_cross_entropy <loss>=0.542818580451\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:34:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=3500 train binary_f_1.000 <score>=0.734878078342\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:34:11 INFO 140009951377216] Iter[2] Batch [4000]#011Speed: 19558.93 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:34:11 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=4000 train binary_classification_accuracy <score>=0.727093226693\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:34:11 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=4000 train binary_classification_cross_entropy <loss>=0.546674393467\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:34:11 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=4000 train binary_f_1.000 <score>=0.731392424868\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:34:18 INFO 140658308478784] Iter[2] Batch [4000]#011Speed: 19187.10 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:34:18 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=4000 train binary_classification_accuracy <score>=0.728032991752\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:34:18 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=4000 train binary_classification_cross_entropy <loss>=0.545633447237\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:34:18 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=4000 train binary_f_1.000 <score>=0.732320809625\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:34:30 INFO 140667034941248] Iter[2] Batch [4000]#011Speed: 19235.67 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:34:30 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=4000 train binary_classification_accuracy <score>=0.729692076981\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:34:30 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=4000 train binary_classification_cross_entropy <loss>=0.54379990084\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:34:30 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=4000 train binary_f_1.000 <score>=0.733783992017\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:34:33 INFO 140684726773568] Iter[2] Batch [4000]#011Speed: 19577.29 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:34:33 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=4000 train binary_classification_accuracy <score>=0.730964008998\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:34:33 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=4000 train binary_classification_cross_entropy <loss>=0.542442336323\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:34:33 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=4000 train binary_f_1.000 <score>=0.735208026711\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:34:35 INFO 140009951377216] Iter[2] Batch [4500]#011Speed: 20063.24 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:34:35 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=4500 train binary_classification_accuracy <score>=0.72738502555\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:34:35 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=4500 train binary_classification_cross_entropy <loss>=0.546310448359\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:34:35 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=4500 train binary_f_1.000 <score>=0.731629230857\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:34:43 INFO 140658308478784] Iter[2] Batch [4500]#011Speed: 20010.66 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:34:43 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=4500 train binary_classification_accuracy <score>=0.728089757832\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:34:43 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=4500 train binary_classification_cross_entropy <loss>=0.545387084819\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:34:43 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=4500 train binary_f_1.000 <score>=0.732402120548\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:34:56 INFO 140684726773568] Iter[2] Batch [4500]#011Speed: 21381.62 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:34:56 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=4500 train binary_classification_accuracy <score>=0.730912241724\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:34:56 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=4500 train binary_classification_cross_entropy <loss>=0.542321878809\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:34:56 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=4500 train binary_f_1.000 <score>=0.735167912732\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:34:57 INFO 140667034941248] Iter[2] Batch [4500]#011Speed: 19004.04 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:34:57 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=4500 train binary_classification_accuracy <score>=0.7298191513\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:34:57 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=4500 train binary_classification_cross_entropy <loss>=0.543550112135\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:34:57 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=4500 train binary_f_1.000 <score>=0.733895440386\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:34:59 INFO 140009951377216] Iter[2] Batch [5000]#011Speed: 20906.76 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:34:59 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=5000 train binary_classification_accuracy <score>=0.727472705459\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:34:59 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=5000 train binary_classification_cross_entropy <loss>=0.54609592653\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:34:59 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=5000 train binary_f_1.000 <score>=0.73179693685\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:35:07 INFO 140658308478784] Iter[2] Batch [5000]#011Speed: 20944.60 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:35:07 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=5000 train binary_classification_accuracy <score>=0.728417716457\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:35:07 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=5000 train binary_classification_cross_entropy <loss>=0.545025907679\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:35:07 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=5000 train binary_f_1.000 <score>=0.732751565839\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:35:22 INFO 140684726773568] Iter[2] Batch [5000]#011Speed: 19316.49 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:35:22 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=5000 train binary_classification_accuracy <score>=0.730852229554\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:35:22 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=5000 train binary_classification_cross_entropy <loss>=0.542220275049\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:35:22 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=5000 train binary_f_1.000 <score>=0.735038942745\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:35:23 INFO 140667034941248] Iter[2] Batch [5000]#011Speed: 18971.58 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:35:23 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=5000 train binary_classification_accuracy <score>=0.730014197161\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:35:23 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=5000 train binary_classification_cross_entropy <loss>=0.543259392671\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:35:23 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=5000 train binary_f_1.000 <score>=0.734146384205\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:35:24 INFO 140009951377216] Iter[2] Batch [5500]#011Speed: 20669.56 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:35:24 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=5500 train binary_classification_accuracy <score>=0.727553535721\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:35:24 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=5500 train binary_classification_cross_entropy <loss>=0.545921002844\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:35:24 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=5500 train binary_f_1.000 <score>=0.731979168194\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:35:31 INFO 140658308478784] Iter[2] Batch [5500]#011Speed: 21248.04 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:35:31 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=5500 train binary_classification_accuracy <score>=0.728487184148\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:35:31 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=5500 train binary_classification_cross_entropy <loss>=0.544835777722\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:35:31 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=5500 train binary_f_1.000 <score>=0.732862967412\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m[08/29/2018 13:35:47 INFO 140684726773568] Iter[2] Batch [5500]#011Speed: 19824.67 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:35:47 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=5500 train binary_classification_accuracy <score>=0.730994001091\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:35:47 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=5500 train binary_classification_cross_entropy <loss>=0.542045264459\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:35:47 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=5500 train binary_f_1.000 <score>=0.735274839373\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:35:48 INFO 140009951377216] Iter[2] Batch [6000]#011Speed: 20433.13 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:35:48 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=6000 train binary_classification_accuracy <score>=0.727723379437\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:35:48 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=6000 train binary_classification_cross_entropy <loss>=0.545751053698\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:35:48 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=6000 train binary_f_1.000 <score>=0.732180667885\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:35:49 INFO 140667034941248] Iter[2] Batch [5500]#011Speed: 19072.96 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:35:49 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=5500 train binary_classification_accuracy <score>=0.730285220869\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:35:49 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=5500 train binary_classification_cross_entropy <loss>=0.542889156417\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:35:49 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=5500 train binary_f_1.000 <score>=0.734473549471\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:35:55 INFO 140658308478784] Iter[2] Batch [6000]#011Speed: 20769.89 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:35:55 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=6000 train binary_classification_accuracy <score>=0.728364939177\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:35:55 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=6000 train binary_classification_cross_entropy <loss>=0.544823269623\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:35:55 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=6000 train binary_f_1.000 <score>=0.732795344644\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:36:10 INFO 140009951377216] Iter[2] Batch [6500]#011Speed: 22522.92 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:36:10 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=6500 train binary_classification_accuracy <score>=0.727571450546\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:36:10 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=6500 train binary_classification_cross_entropy <loss>=0.545720643179\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:36:10 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=6500 train binary_f_1.000 <score>=0.732075597288\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:36:10 INFO 140684726773568] Iter[2] Batch [6000]#011Speed: 22032.45 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:36:10 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=6000 train binary_classification_accuracy <score>=0.730816030662\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:36:10 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=6000 train binary_classification_cross_entropy <loss>=0.542114739167\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:36:10 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=6000 train binary_f_1.000 <score>=0.735167257771\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:36:12 INFO 140667034941248] Iter[2] Batch [6000]#011Speed: 22213.84 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:36:12 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=6000 train binary_classification_accuracy <score>=0.730177803699\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:36:12 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=6000 train binary_classification_cross_entropy <loss>=0.542845934628\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:36:12 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=6000 train binary_f_1.000 <score>=0.734475704457\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:36:19 INFO 140658308478784] Iter[2] Batch [6500]#011Speed: 20552.00 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:36:19 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=6500 train binary_classification_accuracy <score>=0.728547761883\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:36:19 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=6500 train binary_classification_cross_entropy <loss>=0.544601688537\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:36:19 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=6500 train binary_f_1.000 <score>=0.733057533329\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:36:34 INFO 140009951377216] Iter[2] Batch [7000]#011Speed: 20923.95 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:36:34 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=7000 train binary_classification_accuracy <score>=0.727557063277\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:36:34 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=7000 train binary_classification_cross_entropy <loss>=0.545600617811\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:36:34 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=7000 train binary_f_1.000 <score>=0.732056907457\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:36:34 INFO 140684726773568] Iter[2] Batch [6500]#011Speed: 20841.83 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:36:34 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=6500 train binary_classification_accuracy <score>=0.730866174435\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:36:34 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=6500 train binary_classification_cross_entropy <loss>=0.542000044272\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:36:34 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=6500 train binary_f_1.000 <score>=0.735281528688\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:36:34 INFO 140667034941248] Iter[2] Batch [6500]#011Speed: 22106.02 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:36:34 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=6500 train binary_classification_accuracy <score>=0.730329487771\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:36:34 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=6500 train binary_classification_cross_entropy <loss>=0.542616603413\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:36:34 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=6500 train binary_f_1.000 <score>=0.734665769873\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:36:43 INFO 140658308478784] Iter[2] Batch [7000]#011Speed: 20780.59 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:36:43 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=7000 train binary_classification_accuracy <score>=0.728553063848\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:36:43 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=7000 train binary_classification_cross_entropy <loss>=0.544472067637\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:36:43 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=7000 train binary_f_1.000 <score>=0.733108630012\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:36:56 INFO 140684726773568] Iter[2] Batch [7000]#011Speed: 22566.73 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:36:56 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=7000 train binary_classification_accuracy <score>=0.730866733324\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:36:56 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=7000 train binary_classification_cross_entropy <loss>=0.541902286555\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:36:56 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=7000 train binary_f_1.000 <score>=0.735310388347\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:36:57 INFO 140667034941248] Iter[2] Batch [7000]#011Speed: 21922.74 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:36:57 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=7000 train binary_classification_accuracy <score>=0.730456506213\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:36:57 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=7000 train binary_classification_cross_entropy <loss>=0.542392600626\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:36:57 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=7000 train binary_f_1.000 <score>=0.734777351015\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:36:57 INFO 140009951377216] Iter[2] Batch [7500]#011Speed: 21710.98 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:36:57 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=7500 train binary_classification_accuracy <score>=0.727664711372\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:36:57 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=7500 train binary_classification_cross_entropy <loss>=0.545434753577\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:36:57 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=7500 train binary_f_1.000 <score>=0.732144269053\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:37:07 INFO 140658308478784] Iter[2] Batch [7500]#011Speed: 20764.02 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:37:07 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=7500 train binary_classification_accuracy <score>=0.728570990535\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:37:07 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=7500 train binary_classification_cross_entropy <loss>=0.544339584576\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:37:07 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=7500 train binary_f_1.000 <score>=0.733123754827\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34m[08/29/2018 13:37:19 INFO 140684726773568] Iter[2] Batch [7500]#011Speed: 22319.41 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:37:19 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=7500 train binary_classification_accuracy <score>=0.730874016798\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:37:19 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=7500 train binary_classification_cross_entropy <loss>=0.541808286141\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:37:19 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=7500 train binary_f_1.000 <score>=0.735279716144\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:37:20 INFO 140009951377216] Iter[2] Batch [8000]#011Speed: 21721.67 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:37:20 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=8000 train binary_classification_accuracy <score>=0.727679540057\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:37:20 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=8000 train binary_classification_cross_entropy <loss>=0.54532236867\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:37:20 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=8000 train binary_f_1.000 <score>=0.732187137476\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:37:21 INFO 140667034941248] Iter[2] Batch [7500]#011Speed: 21296.00 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:37:21 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=7500 train binary_classification_accuracy <score>=0.730566057859\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:37:21 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=7500 train binary_classification_cross_entropy <loss>=0.542207038032\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:37:21 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=7500 train binary_f_1.000 <score>=0.734850792696\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:37:30 INFO 140658308478784] Iter[2] Batch [8000]#011Speed: 22163.68 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:37:30 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=8000 train binary_classification_accuracy <score>=0.728637795276\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:37:30 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=8000 train binary_classification_cross_entropy <loss>=0.544181029308\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:37:30 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=8000 train binary_f_1.000 <score>=0.733216527885\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:37:43 INFO 140684726773568] Iter[2] Batch [8000]#011Speed: 20972.19 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:37:43 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=8000 train binary_classification_accuracy <score>=0.730736032996\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:37:43 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=8000 train binary_classification_cross_entropy <loss>=0.541825060304\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:37:43 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=8000 train binary_f_1.000 <score>=0.735224508713\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:37:44 INFO 140009951377216] Iter[2] Batch [8500]#011Speed: 21131.01 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:37:44 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=8500 train binary_classification_accuracy <score>=0.727562992589\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:37:44 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=8500 train binary_classification_cross_entropy <loss>=0.545304807897\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:37:44 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=8500 train binary_f_1.000 <score>=0.732121651227\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:37:45 INFO 140667034941248] Iter[2] Batch [8000]#011Speed: 20780.76 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:37:45 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=8000 train binary_classification_accuracy <score>=0.730529308836\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:37:45 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=8000 train binary_classification_cross_entropy <loss>=0.542126867801\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:37:45 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=8000 train binary_f_1.000 <score>=0.734844017392\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:37:53 INFO 140658308478784] Iter[2] Batch [8500]#011Speed: 21433.71 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:37:53 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=8500 train binary_classification_accuracy <score>=0.728491118692\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:37:53 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=8500 train binary_classification_cross_entropy <loss>=0.544203000625\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:37:53 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=8500 train binary_f_1.000 <score>=0.733082100287\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:38:07 INFO 140684726773568] Iter[2] Batch [8500]#011Speed: 20713.57 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:38:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=8500 train binary_classification_accuracy <score>=0.730627455593\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:38:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=8500 train binary_classification_cross_entropy <loss>=0.541834395334\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:38:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=8500 train binary_f_1.000 <score>=0.735146324758\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:38:08 INFO 140667034941248] Iter[2] Batch [8500]#011Speed: 21504.48 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:38:08 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=8500 train binary_classification_accuracy <score>=0.730478061405\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:38:08 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=8500 train binary_classification_cross_entropy <loss>=0.542064771332\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:38:08 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=8500 train binary_f_1.000 <score>=0.734863526588\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:38:08 INFO 140009951377216] Iter[2] Batch [9000]#011Speed: 20385.11 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:38:08 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=9000 train binary_classification_accuracy <score>=0.727577046995\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:38:08 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=9000 train binary_classification_cross_entropy <loss>=0.545216689689\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:38:08 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=9000 train binary_f_1.000 <score>=0.732198636245\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:38:17 INFO 140658308478784] Iter[2] Batch [9000]#011Speed: 20534.21 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:38:17 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=9000 train binary_classification_accuracy <score>=0.72857571381\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:38:17 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=9000 train binary_classification_cross_entropy <loss>=0.544066522977\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:38:17 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=9000 train binary_f_1.000 <score>=0.733150022271\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:38:33 INFO 140009951377216] Iter[2] Batch [9500]#011Speed: 20496.42 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:38:33 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=9500 train binary_classification_accuracy <score>=0.727567940217\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:38:33 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=9500 train binary_classification_cross_entropy <loss>=0.545117764478\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:38:33 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=9500 train binary_f_1.000 <score>=0.732192877058\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:38:33 INFO 140684726773568] Iter[2] Batch [9000]#011Speed: 19108.27 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:38:33 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=9000 train binary_classification_accuracy <score>=0.730606599267\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:38:33 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=9000 train binary_classification_cross_entropy <loss>=0.541768422775\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:38:33 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=9000 train binary_f_1.000 <score>=0.735169359632\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:38:35 INFO 140667034941248] Iter[2] Batch [9000]#011Speed: 18917.38 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:38:35 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=9000 train binary_classification_accuracy <score>=0.730535273858\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:38:35 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=9000 train binary_classification_cross_entropy <loss>=0.541933005294\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:38:35 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=9000 train binary_f_1.000 <score>=0.734930521024\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:38:41 INFO 140658308478784] Iter[2] Batch [9500]#011Speed: 20798.80 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:38:41 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=9500 train binary_classification_accuracy <score>=0.728466477213\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:38:41 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=9500 train binary_classification_cross_entropy <loss>=0.544035372247\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:38:41 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=9500 train binary_f_1.000 <score>=0.733085301089\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31m[08/29/2018 13:38:54 INFO 140009951377216] Iter[2] Batch [10000]#011Speed: 23466.98 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:38:54 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=10000 train binary_classification_accuracy <score>=0.727511048895\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:38:54 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=10000 train binary_classification_cross_entropy <loss>=0.545085834703\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:38:54 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=10000 train binary_f_1.000 <score>=0.732135007726\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:38:56 INFO 140684726773568] Iter[2] Batch [9500]#011Speed: 21211.03 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:38:56 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=9500 train binary_classification_accuracy <score>=0.730500578886\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:38:56 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=9500 train binary_classification_cross_entropy <loss>=0.54174878593\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:38:56 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=9500 train binary_f_1.000 <score>=0.735069891295\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:38:56 INFO 140667034941248] Iter[2] Batch [9500]#011Speed: 23052.96 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:38:56 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=9500 train binary_classification_accuracy <score>=0.730459214819\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:38:56 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=9500 train binary_classification_cross_entropy <loss>=0.541877042179\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:38:56 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=9500 train binary_f_1.000 <score>=0.734869215169\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:39:05 INFO 140658308478784] Iter[2] Batch [10000]#011Speed: 20880.92 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:39:05 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=10000 train binary_classification_accuracy <score>=0.728438856114\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:39:05 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=10000 train binary_classification_cross_entropy <loss>=0.543962725859\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:39:05 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=10000 train binary_f_1.000 <score>=0.733083092867\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:39:18 INFO 140009951377216] Iter[2] Batch [10500]#011Speed: 21365.37 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:39:18 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=10500 train binary_classification_accuracy <score>=0.727472716884\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:39:18 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=10500 train binary_classification_cross_entropy <loss>=0.545075565011\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:39:18 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=10500 train binary_f_1.000 <score>=0.732137995017\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:39:19 INFO 140684726773568] Iter[2] Batch [10000]#011Speed: 21981.19 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:39:19 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=10000 train binary_classification_accuracy <score>=0.730441155884\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:39:19 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=10000 train binary_classification_cross_entropy <loss>=0.541714793145\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:39:19 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=10000 train binary_f_1.000 <score>=0.735009245676\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:39:19 INFO 140667034941248] Iter[2] Batch [10000]#011Speed: 21762.58 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:39:19 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=10000 train binary_classification_accuracy <score>=0.730401459854\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:39:19 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=10000 train binary_classification_cross_entropy <loss>=0.541839090896\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:39:19 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=10000 train binary_f_1.000 <score>=0.734792459808\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:39:29 INFO 140658308478784] Iter[2] Batch [10500]#011Speed: 21563.09 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:39:29 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=10500 train binary_classification_accuracy <score>=0.728409960956\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:39:29 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=10500 train binary_classification_cross_entropy <loss>=0.543917821012\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:39:29 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=10500 train binary_f_1.000 <score>=0.733080630517\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:39:41 INFO 140009951377216] Iter[2] Batch [11000]#011Speed: 21416.51 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:39:41 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=11000 train binary_classification_accuracy <score>=0.727430324516\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:39:41 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=11000 train binary_classification_cross_entropy <loss>=0.54501373159\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:39:41 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=11000 train binary_f_1.000 <score>=0.7321132173\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:39:44 INFO 140684726773568] Iter[2] Batch [10500]#011Speed: 20161.45 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:39:44 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=10500 train binary_classification_accuracy <score>=0.730328159223\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:39:44 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=10500 train binary_classification_cross_entropy <loss>=0.541736304759\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:39:44 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=10500 train binary_f_1.000 <score>=0.734896145289\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:39:44 INFO 140667034941248] Iter[2] Batch [10500]#011Speed: 20210.46 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:39:44 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=10500 train binary_classification_accuracy <score>=0.730333015903\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:39:44 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=10500 train binary_classification_cross_entropy <loss>=0.541831167522\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:39:44 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=10500 train binary_f_1.000 <score>=0.734762489899\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:39:52 INFO 140658308478784] Iter[2] Batch [11000]#011Speed: 21305.80 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:39:52 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=11000 train binary_classification_accuracy <score>=0.728344695937\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:39:52 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=11000 train binary_classification_cross_entropy <loss>=0.543888148052\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:39:52 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=11000 train binary_f_1.000 <score>=0.733003077476\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:40:04 INFO 140009951377216] Iter[2] Batch [11500]#011Speed: 21484.69 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:40:04 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=11500 train binary_classification_accuracy <score>=0.727457264586\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:40:04 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=11500 train binary_classification_cross_entropy <loss>=0.544942694166\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:40:04 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=11500 train binary_f_1.000 <score>=0.732129086113\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:40:07 INFO 140667034941248] Iter[2] Batch [11000]#011Speed: 21613.77 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:40:07 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=11000 train binary_classification_accuracy <score>=0.730305790383\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:40:07 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=11000 train binary_classification_cross_entropy <loss>=0.541790714841\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:40:07 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=11000 train binary_f_1.000 <score>=0.734749157911\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:40:08 INFO 140684726773568] Iter[2] Batch [11000]#011Speed: 20499.69 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:40:08 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=11000 train binary_classification_accuracy <score>=0.73022852468\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:40:08 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=11000 train binary_classification_cross_entropy <loss>=0.541708656921\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:40:08 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=11000 train binary_f_1.000 <score>=0.73484199912\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:40:17 INFO 140658308478784] Iter[2] Batch [11500]#011Speed: 19729.09 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:40:17 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=11500 train binary_classification_accuracy <score>=0.728404834362\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:40:17 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=11500 train binary_classification_cross_entropy <loss>=0.543797777241\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:40:17 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=11500 train binary_f_1.000 <score>=0.733051470312\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31m[08/29/2018 13:40:30 INFO 140009951377216] Iter[2] Batch [12000]#011Speed: 19705.24 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:40:30 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=12000 train binary_classification_accuracy <score>=0.727349304225\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:40:30 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=12000 train binary_classification_cross_entropy <loss>=0.544961277056\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:40:30 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=12000 train binary_f_1.000 <score>=0.73201539825\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:40:32 INFO 140667034941248] Iter[2] Batch [11500]#011Speed: 19908.76 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:40:32 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=11500 train binary_classification_accuracy <score>=0.730248847926\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:40:32 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=11500 train binary_classification_cross_entropy <loss>=0.541775071335\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:40:32 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=11500 train binary_f_1.000 <score>=0.734713893811\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:40:33 INFO 140684726773568] Iter[2] Batch [11500]#011Speed: 19995.58 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:40:33 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=11500 train binary_classification_accuracy <score>=0.730249195722\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:40:33 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=11500 train binary_classification_cross_entropy <loss>=0.541632791209\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:40:33 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=11500 train binary_f_1.000 <score>=0.734836222433\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:40:42 INFO 140658308478784] Iter[2] Batch [12000]#011Speed: 20550.75 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:40:42 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=12000 train binary_classification_accuracy <score>=0.7284047996\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:40:42 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=12000 train binary_classification_cross_entropy <loss>=0.543734368404\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:40:42 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=12000 train binary_f_1.000 <score>=0.733072177484\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:40:53 INFO 140009951377216] Iter[2] Batch [12500]#011Speed: 21710.69 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:40:53 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=12500 train binary_classification_accuracy <score>=0.727368290537\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:40:53 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=12500 train binary_classification_cross_entropy <loss>=0.544853392983\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:40:53 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=12500 train binary_f_1.000 <score>=0.732027277453\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:40:57 INFO 140667034941248] Iter[2] Batch [12000]#011Speed: 20400.19 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:40:57 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=12000 train binary_classification_accuracy <score>=0.730197566869\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:40:57 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=12000 train binary_classification_cross_entropy <loss>=0.541737763611\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:40:57 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=12000 train binary_f_1.000 <score>=0.734658840361\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:40:57 INFO 140684726773568] Iter[2] Batch [12000]#011Speed: 21155.38 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:40:57 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=12000 train binary_classification_accuracy <score>=0.730238563453\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:40:57 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=12000 train binary_classification_cross_entropy <loss>=0.541576999427\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:40:57 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=12000 train binary_f_1.000 <score>=0.734760889187\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:41:05 INFO 140658308478784] Iter[2] Batch [12500]#011Speed: 21363.18 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:41:05 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=12500 train binary_classification_accuracy <score>=0.728367010639\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:41:05 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=12500 train binary_classification_cross_entropy <loss>=0.543670092779\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:41:05 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=12500 train binary_f_1.000 <score>=0.733030698037\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:41:16 INFO 140009951377216] Iter[2] Batch [13000]#011Speed: 21239.80 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:41:16 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=13000 train binary_classification_accuracy <score>=0.727325128836\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:41:16 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=13000 train binary_classification_cross_entropy <loss>=0.544776382983\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:41:16 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=13000 train binary_f_1.000 <score>=0.731964345445\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:41:21 INFO 140684726773568] Iter[2] Batch [12500]#011Speed: 21095.65 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:41:21 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=12500 train binary_classification_accuracy <score>=0.730187984961\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:41:21 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=12500 train binary_classification_cross_entropy <loss>=0.541531543102\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:41:21 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=12500 train binary_f_1.000 <score>=0.734709572846\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:41:21 INFO 140667034941248] Iter[2] Batch [12500]#011Speed: 20751.79 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:41:21 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=12500 train binary_classification_accuracy <score>=0.730183985281\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:41:21 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=12500 train binary_classification_cross_entropy <loss>=0.541672133141\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:41:21 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=12500 train binary_f_1.000 <score>=0.734634425577\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:41:30 INFO 140658308478784] Iter[2] Batch [13000]#011Speed: 20142.84 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:41:30 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=13000 train binary_classification_accuracy <score>=0.728325513422\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:41:30 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=13000 train binary_classification_cross_entropy <loss>=0.543594383942\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:41:30 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=13000 train binary_f_1.000 <score>=0.732972018022\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:41:39 INFO 140009951377216] Iter[2] Batch [13500]#011Speed: 21712.87 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:41:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=13500 train binary_classification_accuracy <score>=0.727322124287\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:41:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=13500 train binary_classification_cross_entropy <loss>=0.544735289422\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:41:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=13500 train binary_f_1.000 <score>=0.731970327778\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:41:45 INFO 140684726773568] Iter[2] Batch [13000]#011Speed: 20189.82 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:41:45 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=13000 train binary_classification_accuracy <score>=0.730054534267\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:41:45 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=13000 train binary_classification_cross_entropy <loss>=0.541516616015\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:41:45 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=13000 train binary_f_1.000 <score>=0.734557180321\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:41:47 INFO 140667034941248] Iter[2] Batch [13000]#011Speed: 18971.26 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:41:47 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=13000 train binary_classification_accuracy <score>=0.730102453657\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:41:47 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=13000 train binary_classification_cross_entropy <loss>=0.541616990958\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:41:47 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=13000 train binary_f_1.000 <score>=0.734598298464\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:41:55 INFO 140658308478784] Iter[2] Batch [13500]#011Speed: 19678.13 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:41:55 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=13500 train binary_classification_accuracy <score>=0.728354047848\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:41:55 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=13500 train binary_classification_cross_entropy <loss>=0.54351404769\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:41:55 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=13500 train binary_f_1.000 <score>=0.733017083158\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31m[08/29/2018 13:42:03 INFO 140009951377216] Iter[2] Batch [14000]#011Speed: 20802.03 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=14000 train binary_classification_accuracy <score>=0.727299764303\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=14000 train binary_classification_cross_entropy <loss>=0.544646367671\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, batch=14000 train binary_f_1.000 <score>=0.731932143814\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, train binary_classification_accuracy <score>=0.727299764303\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, train binary_classification_cross_entropy <loss>=0.544646367671\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=2, train binary_f_1.000 <score>=0.731932143814\u001b[0m\n",
      "\u001b[31m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 661857.7129840851, \"sum\": 661857.7129840851, \"min\": 661857.7129840851}}, \"EndTime\": 1535550123.640554, \"Dimensions\": {\"Host\": \"algo-1\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535549461.771749}\n",
      "\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:03 INFO 140009951377216] #progress_metric: host=algo-1, completed 3 % of epochs\u001b[0m\n",
      "\u001b[31m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Total Batches Seen\": {\"count\": 1, \"max\": 42004, \"sum\": 42004.0, \"min\": 42004}, \"Total Records Seen\": {\"count\": 1, \"max\": 42001552, \"sum\": 42001552.0, \"min\": 42001552}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Reset Count\": {\"count\": 1, \"max\": 4, \"sum\": 4.0, \"min\": 4}}, \"EndTime\": 1535550123.640784, \"Dimensions\": {\"Host\": \"algo-1\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 2}, \"StartTime\": 1535549461.782806}\n",
      "\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:03 INFO 140009951377216] #throughput_metric: host=algo-1, train throughput=21152.8468534 records/second\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=0 train binary_classification_accuracy <score>=0.722\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=0 train binary_classification_cross_entropy <loss>=0.536569702148\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=0 train binary_f_1.000 <score>=0.750896057348\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:08 INFO 140684726773568] Iter[2] Batch [13500]#011Speed: 22528.08 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:08 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=13500 train binary_classification_accuracy <score>=0.730114584105\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:08 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=13500 train binary_classification_cross_entropy <loss>=0.541420356628\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:08 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=13500 train binary_f_1.000 <score>=0.73462992444\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:10 INFO 140667034941248] Iter[2] Batch [13500]#011Speed: 21718.98 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:10 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=13500 train binary_classification_accuracy <score>=0.730176505444\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:10 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=13500 train binary_classification_cross_entropy <loss>=0.541495500003\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:10 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=13500 train binary_f_1.000 <score>=0.734717355987\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:42:17 INFO 140658308478784] Iter[2] Batch [14000]#011Speed: 22645.59 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:42:17 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=14000 train binary_classification_accuracy <score>=0.728354331833\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:42:17 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=14000 train binary_classification_cross_entropy <loss>=0.543411606398\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:42:17 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, batch=14000 train binary_f_1.000 <score>=0.733047011119\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:42:17 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, train binary_classification_accuracy <score>=0.728354331833\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:42:17 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, train binary_classification_cross_entropy <loss>=0.543411606398\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:42:17 INFO 140658308478784] #quality_metric: host=algo-2, epoch=2, train binary_f_1.000 <score>=0.733047011119\u001b[0m\n",
      "\u001b[32m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 675707.7970504761, \"sum\": 675707.7970504761, \"min\": 675707.7970504761}}, \"EndTime\": 1535550137.919564, \"Dimensions\": {\"Host\": \"algo-2\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535549462.203896}\n",
      "\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:42:17 INFO 140658308478784] #progress_metric: host=algo-2, completed 3 % of epochs\u001b[0m\n",
      "\u001b[32m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Total Batches Seen\": {\"count\": 1, \"max\": 42004, \"sum\": 42004.0, \"min\": 42004}, \"Total Records Seen\": {\"count\": 1, \"max\": 42001552, \"sum\": 42001552.0, \"min\": 42001552}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Reset Count\": {\"count\": 1, \"max\": 4, \"sum\": 4.0, \"min\": 4}}, \"EndTime\": 1535550137.919832, \"Dimensions\": {\"Host\": \"algo-2\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 2}, \"StartTime\": 1535549462.211722}\n",
      "\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:42:17 INFO 140658308478784] #throughput_metric: host=algo-2, train throughput=20719.2722142 records/second\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:42:17 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=0 train binary_classification_accuracy <score>=0.729\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:42:17 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=0 train binary_classification_cross_entropy <loss>=0.534887817383\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:42:17 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=0 train binary_f_1.000 <score>=0.752058554437\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:25 INFO 140009951377216] Iter[3] Batch [500]#011Speed: 22413.90 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:25 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=500 train binary_classification_accuracy <score>=0.728968063872\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:25 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=500 train binary_classification_cross_entropy <loss>=0.54069411933\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:25 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=500 train binary_f_1.000 <score>=0.734280855627\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:30 INFO 140684726773568] Iter[2] Batch [14000]#011Speed: 22007.67 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:30 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=14000 train binary_classification_accuracy <score>=0.730122134133\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:30 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=14000 train binary_classification_cross_entropy <loss>=0.541344618093\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:30 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, batch=14000 train binary_f_1.000 <score>=0.73464631869\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:30 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, train binary_classification_accuracy <score>=0.730122134133\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:30 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, train binary_classification_cross_entropy <loss>=0.541344618093\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:30 INFO 140684726773568] #quality_metric: host=algo-4, epoch=2, train binary_f_1.000 <score>=0.73464631869\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 674267.4739360809, \"sum\": 674267.4739360809, \"min\": 674267.4739360809}}, \"EndTime\": 1535550150.898686, \"Dimensions\": {\"Host\": \"algo-4\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535549476.628457}\n",
      "\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:30 INFO 140684726773568] #progress_metric: host=algo-4, completed 3 % of epochs\u001b[0m\n",
      "\u001b[34m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Total Batches Seen\": {\"count\": 1, \"max\": 42004, \"sum\": 42004.0, \"min\": 42004}, \"Total Records Seen\": {\"count\": 1, \"max\": 42001552, \"sum\": 42001552.0, \"min\": 42001552}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Reset Count\": {\"count\": 1, \"max\": 4, \"sum\": 4.0, \"min\": 4}}, \"EndTime\": 1535550150.898902, \"Dimensions\": {\"Host\": \"algo-4\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 2}, \"StartTime\": 1535549476.631187}\n",
      "\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:30 INFO 140684726773568] #throughput_metric: host=algo-4, train throughput=20763.5348499 records/second\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:31 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=0 train binary_classification_accuracy <score>=0.711\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:31 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=0 train binary_classification_cross_entropy <loss>=0.55133190918\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:31 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=0 train binary_f_1.000 <score>=0.682068206821\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:32 INFO 140667034941248] Iter[2] Batch [14000]#011Speed: 22653.23 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:32 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=14000 train binary_classification_accuracy <score>=0.730133561888\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:32 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=14000 train binary_classification_cross_entropy <loss>=0.541415306844\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:32 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, batch=14000 train binary_f_1.000 <score>=0.734681031604\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:32 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, train binary_classification_accuracy <score>=0.730133561888\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:32 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, train binary_classification_cross_entropy <loss>=0.541415306844\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:32 INFO 140667034941248] #quality_metric: host=algo-3, epoch=2, train binary_f_1.000 <score>=0.734681031604\u001b[0m\n",
      "\u001b[33m#metrics {\"Metrics\": {\"update.time\": {\"count\": 1, \"max\": 680024.5668888092, \"sum\": 680024.5668888092, \"min\": 680024.5668888092}}, \"EndTime\": 1535550152.726674, \"Dimensions\": {\"Host\": \"algo-3\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\"}, \"StartTime\": 1535549472.696329}\n",
      "\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:32 INFO 140667034941248] #progress_metric: host=algo-3, completed 3 % of epochs\u001b[0m\n",
      "\u001b[33m#metrics {\"Metrics\": {\"Max Batches Seen Between Resets\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Batches Since Last Reset\": {\"count\": 1, \"max\": 14001, \"sum\": 14001.0, \"min\": 14001}, \"Number of Records Since Last Reset\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Total Batches Seen\": {\"count\": 1, \"max\": 42004, \"sum\": 42004.0, \"min\": 42004}, \"Total Records Seen\": {\"count\": 1, \"max\": 42001552, \"sum\": 42001552.0, \"min\": 42001552}, \"Max Records Seen Between Resets\": {\"count\": 1, \"max\": 14000184, \"sum\": 14000184.0, \"min\": 14000184}, \"Reset Count\": {\"count\": 1, \"max\": 4, \"sum\": 4.0, \"min\": 4}}, \"EndTime\": 1535550152.726845, \"Dimensions\": {\"Host\": \"algo-3\", \"Meta\": \"training_data_iter\", \"Operation\": \"training\", \"Algorithm\": \"factorization-machines\", \"epoch\": 2}, \"StartTime\": 1535549472.702043}\n",
      "\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:32 INFO 140667034941248] #throughput_metric: host=algo-3, train throughput=20587.7514961 records/second\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:32 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=0 train binary_classification_accuracy <score>=0.73\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:32 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=0 train binary_classification_cross_entropy <loss>=0.537618896484\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:32 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=0 train binary_f_1.000 <score>=0.768835616438\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:42:41 INFO 140658308478784] Iter[3] Batch [500]#011Speed: 21089.75 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:42:41 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=500 train binary_classification_accuracy <score>=0.730966067864\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:42:41 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=500 train binary_classification_cross_entropy <loss>=0.538610518648\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:42:41 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=500 train binary_f_1.000 <score>=0.73524443327\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31m[08/29/2018 13:42:49 INFO 140009951377216] Iter[3] Batch [1000]#011Speed: 21607.65 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:49 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=1000 train binary_classification_accuracy <score>=0.729478521479\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:49 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=1000 train binary_classification_cross_entropy <loss>=0.540287957678\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:42:49 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=1000 train binary_f_1.000 <score>=0.734409848703\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:54 INFO 140667034941248] Iter[3] Batch [500]#011Speed: 22960.79 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:54 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=500 train binary_classification_accuracy <score>=0.733157684631\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:54 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=500 train binary_classification_cross_entropy <loss>=0.536878102011\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:42:54 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=500 train binary_f_1.000 <score>=0.738247490905\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:54 INFO 140684726773568] Iter[3] Batch [500]#011Speed: 21117.32 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:54 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=500 train binary_classification_accuracy <score>=0.731111776447\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:54 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=500 train binary_classification_cross_entropy <loss>=0.538034479381\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:42:54 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=500 train binary_f_1.000 <score>=0.736348495257\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:43:06 INFO 140658308478784] Iter[3] Batch [1000]#011Speed: 19886.67 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:43:06 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=1000 train binary_classification_accuracy <score>=0.730905094905\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:43:06 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=1000 train binary_classification_cross_entropy <loss>=0.538571338452\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:43:06 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=1000 train binary_f_1.000 <score>=0.735246939305\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:43:15 INFO 140009951377216] Iter[3] Batch [1500]#011Speed: 19187.23 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:43:15 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=1500 train binary_classification_accuracy <score>=0.729947368421\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:43:15 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=1500 train binary_classification_cross_entropy <loss>=0.539632447387\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:43:15 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=1500 train binary_f_1.000 <score>=0.73481764293\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:43:19 INFO 140667034941248] Iter[3] Batch [1000]#011Speed: 20354.20 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:43:19 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=1000 train binary_classification_accuracy <score>=0.731949050949\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:43:19 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=1000 train binary_classification_cross_entropy <loss>=0.53724343241\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:43:19 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=1000 train binary_f_1.000 <score>=0.736679094115\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:43:20 INFO 140684726773568] Iter[3] Batch [1000]#011Speed: 19274.50 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:43:20 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=1000 train binary_classification_accuracy <score>=0.731733266733\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:43:20 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=1000 train binary_classification_cross_entropy <loss>=0.537290521338\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:43:20 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=1000 train binary_f_1.000 <score>=0.736404515779\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:43:31 INFO 140658308478784] Iter[3] Batch [1500]#011Speed: 20377.99 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:43:31 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=1500 train binary_classification_accuracy <score>=0.730764823451\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:43:31 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=1500 train binary_classification_cross_entropy <loss>=0.538257403329\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:43:31 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=1500 train binary_f_1.000 <score>=0.735159308634\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:43:39 INFO 140009951377216] Iter[3] Batch [2000]#011Speed: 20741.17 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:43:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=2000 train binary_classification_accuracy <score>=0.730611694153\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:43:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=2000 train binary_classification_cross_entropy <loss>=0.539003574644\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:43:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=2000 train binary_f_1.000 <score>=0.735393650579\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:43:41 INFO 140667034941248] Iter[3] Batch [1500]#011Speed: 21855.09 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:43:41 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=1500 train binary_classification_accuracy <score>=0.732818787475\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:43:41 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=1500 train binary_classification_cross_entropy <loss>=0.536405569566\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:43:41 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=1500 train binary_f_1.000 <score>=0.737560703635\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:43:44 INFO 140684726773568] Iter[3] Batch [1500]#011Speed: 21006.96 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:43:44 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=1500 train binary_classification_accuracy <score>=0.732582944704\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:43:44 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=1500 train binary_classification_cross_entropy <loss>=0.536274422642\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:43:44 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=1500 train binary_f_1.000 <score>=0.737359296001\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:43:55 INFO 140658308478784] Iter[3] Batch [2000]#011Speed: 20867.09 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:43:55 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=2000 train binary_classification_accuracy <score>=0.731104447776\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:43:55 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=2000 train binary_classification_cross_entropy <loss>=0.53786015804\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:43:55 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=2000 train binary_f_1.000 <score>=0.735477203922\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:44:03 INFO 140009951377216] Iter[3] Batch [2500]#011Speed: 20795.09 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:44:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=2500 train binary_classification_accuracy <score>=0.730645341863\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:44:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=2500 train binary_classification_cross_entropy <loss>=0.538755756987\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:44:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=2500 train binary_f_1.000 <score>=0.735368293032\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:44:06 INFO 140667034941248] Iter[3] Batch [2000]#011Speed: 20101.59 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:44:06 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=2000 train binary_classification_accuracy <score>=0.732777611194\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:44:06 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=2000 train binary_classification_cross_entropy <loss>=0.536241039575\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:44:06 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=2000 train binary_f_1.000 <score>=0.737410266257\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:44:06 INFO 140684726773568] Iter[3] Batch [2000]#011Speed: 22324.04 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:44:06 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=2000 train binary_classification_accuracy <score>=0.733064467766\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:44:06 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=2000 train binary_classification_cross_entropy <loss>=0.535752318101\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:44:06 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=2000 train binary_f_1.000 <score>=0.737711213141\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:44:23 INFO 140658308478784] Iter[3] Batch [2500]#011Speed: 17948.23 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:44:23 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=2500 train binary_classification_accuracy <score>=0.73118992403\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:44:23 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=2500 train binary_classification_cross_entropy <loss>=0.53765122208\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:44:23 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=2500 train binary_f_1.000 <score>=0.735657429732\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:44:28 INFO 140009951377216] Iter[3] Batch [3000]#011Speed: 19936.15 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:44:28 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=3000 train binary_classification_accuracy <score>=0.730657114295\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:44:28 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=3000 train binary_classification_cross_entropy <loss>=0.538553684488\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:44:28 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=3000 train binary_f_1.000 <score>=0.735441384886\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:44:31 INFO 140684726773568] Iter[3] Batch [2500]#011Speed: 20655.49 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:44:31 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=2500 train binary_classification_accuracy <score>=0.733179928029\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:44:31 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=2500 train binary_classification_cross_entropy <loss>=0.535583574968\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:44:31 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=2500 train binary_f_1.000 <score>=0.737756965493\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:44:33 INFO 140667034941248] Iter[3] Batch [2500]#011Speed: 18839.00 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:44:33 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=2500 train binary_classification_accuracy <score>=0.733054378249\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:44:33 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=2500 train binary_classification_cross_entropy <loss>=0.535823709237\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:44:33 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=2500 train binary_f_1.000 <score>=0.737567978176\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[08/29/2018 13:44:46 INFO 140658308478784] Iter[3] Batch [3000]#011Speed: 21893.27 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:44:46 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=3000 train binary_classification_accuracy <score>=0.731613462179\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:44:46 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=3000 train binary_classification_cross_entropy <loss>=0.537194429972\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:44:46 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=3000 train binary_f_1.000 <score>=0.736197006232\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:44:50 INFO 140009951377216] Iter[3] Batch [3500]#011Speed: 22471.47 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:44:50 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=3500 train binary_classification_accuracy <score>=0.730838046272\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:44:50 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=3500 train binary_classification_cross_entropy <loss>=0.538389810614\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:44:50 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=3500 train binary_f_1.000 <score>=0.735634531438\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:44:55 INFO 140684726773568] Iter[3] Batch [3000]#011Speed: 20912.76 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:44:55 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=3000 train binary_classification_accuracy <score>=0.733371209597\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:44:55 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=3000 train binary_classification_cross_entropy <loss>=0.535339896813\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:44:55 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=3000 train binary_f_1.000 <score>=0.737910398873\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:44:56 INFO 140667034941248] Iter[3] Batch [3000]#011Speed: 21312.00 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:44:56 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=3000 train binary_classification_accuracy <score>=0.733303232256\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:44:56 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=3000 train binary_classification_cross_entropy <loss>=0.535419828319\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:44:56 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=3000 train binary_f_1.000 <score>=0.737787580148\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:45:10 INFO 140658308478784] Iter[3] Batch [3500]#011Speed: 20811.54 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:45:10 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=3500 train binary_classification_accuracy <score>=0.731503284776\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:45:10 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=3500 train binary_classification_cross_entropy <loss>=0.537220232147\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:45:10 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=3500 train binary_f_1.000 <score>=0.73602864798\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:45:15 INFO 140009951377216] Iter[3] Batch [4000]#011Speed: 20264.91 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:45:15 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=4000 train binary_classification_accuracy <score>=0.731040489878\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:45:15 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=4000 train binary_classification_cross_entropy <loss>=0.538156631649\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:45:15 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=4000 train binary_f_1.000 <score>=0.735892451177\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:45:18 INFO 140684726773568] Iter[3] Batch [3500]#011Speed: 20907.82 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:45:18 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=3500 train binary_classification_accuracy <score>=0.733287346472\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:45:18 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=3500 train binary_classification_cross_entropy <loss>=0.535340419818\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:45:18 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=3500 train binary_f_1.000 <score>=0.737825047486\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:45:21 INFO 140667034941248] Iter[3] Batch [3500]#011Speed: 20564.19 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:45:21 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=3500 train binary_classification_accuracy <score>=0.733303913168\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:45:21 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=3500 train binary_classification_cross_entropy <loss>=0.535299120043\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:45:21 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=3500 train binary_f_1.000 <score>=0.737876217446\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:45:34 INFO 140658308478784] Iter[3] Batch [4000]#011Speed: 20389.25 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:45:34 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=4000 train binary_classification_accuracy <score>=0.731967258185\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:45:34 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=4000 train binary_classification_cross_entropy <loss>=0.536777933462\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:45:34 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=4000 train binary_f_1.000 <score>=0.736647895895\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:45:39 INFO 140009951377216] Iter[3] Batch [4500]#011Speed: 20490.69 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:45:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=4500 train binary_classification_accuracy <score>=0.731203065985\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:45:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=4500 train binary_classification_cross_entropy <loss>=0.537908081461\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:45:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=4500 train binary_f_1.000 <score>=0.736028737479\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:45:42 INFO 140684726773568] Iter[3] Batch [4000]#011Speed: 21339.18 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:45:42 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=4000 train binary_classification_accuracy <score>=0.733524618845\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:45:42 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=4000 train binary_classification_cross_entropy <loss>=0.535007722274\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:45:42 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=4000 train binary_f_1.000 <score>=0.73815574773\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:45:44 INFO 140667034941248] Iter[3] Batch [4000]#011Speed: 21254.70 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:45:44 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=4000 train binary_classification_accuracy <score>=0.733576105974\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:45:44 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=4000 train binary_classification_cross_entropy <loss>=0.534986158154\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:45:44 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=4000 train binary_f_1.000 <score>=0.73832959961\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:45:57 INFO 140658308478784] Iter[3] Batch [4500]#011Speed: 21443.88 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:45:57 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=4500 train binary_classification_accuracy <score>=0.732059097978\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:45:57 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=4500 train binary_classification_cross_entropy <loss>=0.536627420883\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:45:57 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=4500 train binary_f_1.000 <score>=0.736766078415\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:46:03 INFO 140009951377216] Iter[3] Batch [5000]#011Speed: 20741.06 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:46:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=5000 train binary_classification_accuracy <score>=0.731159768046\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:46:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=5000 train binary_classification_cross_entropy <loss>=0.537800445654\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:46:03 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=5000 train binary_f_1.000 <score>=0.73595181234\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:46:05 INFO 140684726773568] Iter[3] Batch [4500]#011Speed: 21213.84 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:46:05 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=4500 train binary_classification_accuracy <score>=0.733535880915\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:46:05 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=4500 train binary_classification_cross_entropy <loss>=0.534945845632\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:46:05 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=4500 train binary_f_1.000 <score>=0.738079654813\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33m[08/29/2018 13:46:09 INFO 140667034941248] Iter[3] Batch [4500]#011Speed: 19838.96 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:46:09 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=4500 train binary_classification_accuracy <score>=0.733822706065\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:46:09 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=4500 train binary_classification_cross_entropy <loss>=0.534671493534\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:46:09 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=4500 train binary_f_1.000 <score>=0.738614065303\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:46:22 INFO 140658308478784] Iter[3] Batch [5000]#011Speed: 20267.91 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:46:22 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=5000 train binary_classification_accuracy <score>=0.732214157169\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:46:22 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=5000 train binary_classification_cross_entropy <loss>=0.536438374209\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:46:22 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=5000 train binary_f_1.000 <score>=0.737043027295\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:46:27 INFO 140009951377216] Iter[3] Batch [5500]#011Speed: 21005.60 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:46:27 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=5500 train binary_classification_accuracy <score>=0.731218869296\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:46:27 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=5500 train binary_classification_cross_entropy <loss>=0.537663177601\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:46:27 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=5500 train binary_f_1.000 <score>=0.736065787686\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:46:28 INFO 140684726773568] Iter[3] Batch [5000]#011Speed: 21681.07 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:46:28 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=5000 train binary_classification_accuracy <score>=0.73349130174\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:46:28 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=5000 train binary_classification_cross_entropy <loss>=0.534875956474\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:46:29 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=5000 train binary_f_1.000 <score>=0.738080883035\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:46:31 INFO 140667034941248] Iter[3] Batch [5000]#011Speed: 23406.61 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:46:31 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=5000 train binary_classification_accuracy <score>=0.733868226355\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:46:31 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=5000 train binary_classification_cross_entropy <loss>=0.534491239405\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:46:31 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=5000 train binary_f_1.000 <score>=0.738674658319\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:46:46 INFO 140658308478784] Iter[3] Batch [5500]#011Speed: 21011.26 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:46:46 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=5500 train binary_classification_accuracy <score>=0.732308671151\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:46:46 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=5500 train binary_classification_cross_entropy <loss>=0.536303059483\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:46:46 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=5500 train binary_f_1.000 <score>=0.737104917377\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:46:51 INFO 140009951377216] Iter[3] Batch [6000]#011Speed: 21129.42 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:46:51 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=6000 train binary_classification_accuracy <score>=0.731253957674\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:46:51 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=6000 train binary_classification_cross_entropy <loss>=0.537613930159\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:46:51 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=6000 train binary_f_1.000 <score>=0.736154122135\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:46:52 INFO 140684726773568] Iter[3] Batch [5500]#011Speed: 21140.65 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:46:52 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=5500 train binary_classification_accuracy <score>=0.733592983094\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:46:52 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=5500 train binary_classification_cross_entropy <loss>=0.534701232439\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:46:52 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=5500 train binary_f_1.000 <score>=0.738235881978\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:46:54 INFO 140667034941248] Iter[3] Batch [5500]#011Speed: 21229.56 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:46:54 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=5500 train binary_classification_accuracy <score>=0.733987456826\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:46:54 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=5500 train binary_classification_cross_entropy <loss>=0.534300841198\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:46:54 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=5500 train binary_f_1.000 <score>=0.73882212004\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:47:09 INFO 140658308478784] Iter[3] Batch [6000]#011Speed: 21214.19 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:47:09 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=6000 train binary_classification_accuracy <score>=0.732231128145\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:47:09 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=6000 train binary_classification_cross_entropy <loss>=0.536284487909\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:47:09 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=6000 train binary_f_1.000 <score>=0.737098077533\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:47:14 INFO 140009951377216] Iter[3] Batch [6500]#011Speed: 21612.17 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:47:14 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=6500 train binary_classification_accuracy <score>=0.731274880788\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:47:14 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=6500 train binary_classification_cross_entropy <loss>=0.537509511828\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:47:14 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=6500 train binary_f_1.000 <score>=0.736238832687\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:47:16 INFO 140684726773568] Iter[3] Batch [6000]#011Speed: 20802.35 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:47:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=6000 train binary_classification_accuracy <score>=0.73352107982\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:47:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=6000 train binary_classification_cross_entropy <loss>=0.534700494447\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:47:16 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=6000 train binary_f_1.000 <score>=0.738220203833\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:47:17 INFO 140667034941248] Iter[3] Batch [6000]#011Speed: 21954.09 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:47:17 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=6000 train binary_classification_accuracy <score>=0.733908681886\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:47:17 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=6000 train binary_classification_cross_entropy <loss>=0.534297553415\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:47:17 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=6000 train binary_f_1.000 <score>=0.738814409703\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:47:34 INFO 140658308478784] Iter[3] Batch [6500]#011Speed: 20475.36 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:47:34 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=6500 train binary_classification_accuracy <score>=0.732258575604\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:47:34 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=6500 train binary_classification_cross_entropy <loss>=0.536198335379\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:47:34 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=6500 train binary_f_1.000 <score>=0.737149789043\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:47:39 INFO 140009951377216] Iter[3] Batch [7000]#011Speed: 20149.54 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:47:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=7000 train binary_classification_accuracy <score>=0.731196400514\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:47:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=7000 train binary_classification_cross_entropy <loss>=0.537486625367\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:47:39 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=7000 train binary_f_1.000 <score>=0.736210251991\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:47:41 INFO 140667034941248] Iter[3] Batch [6500]#011Speed: 20577.58 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:47:41 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=6500 train binary_classification_accuracy <score>=0.73396723581\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:47:41 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=6500 train binary_classification_cross_entropy <loss>=0.534208632698\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:47:41 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=6500 train binary_f_1.000 <score>=0.738871321563\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:47:42 INFO 140684726773568] Iter[3] Batch [6500]#011Speed: 19611.19 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:47:42 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=6500 train binary_classification_accuracy <score>=0.73354945393\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:47:42 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=6500 train binary_classification_cross_entropy <loss>=0.534631538437\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:47:42 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=6500 train binary_f_1.000 <score>=0.738326672644\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[32m[08/29/2018 13:48:01 INFO 140658308478784] Iter[3] Batch [7000]#011Speed: 18506.97 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:48:01 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=7000 train binary_classification_accuracy <score>=0.732130267105\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:48:01 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=7000 train binary_classification_cross_entropy <loss>=0.53621455598\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:48:01 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=7000 train binary_f_1.000 <score>=0.737054384778\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:48:04 INFO 140009951377216] Iter[3] Batch [7500]#011Speed: 19724.35 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:48:04 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=7500 train binary_classification_accuracy <score>=0.73111105186\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:48:04 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=7500 train binary_classification_cross_entropy <loss>=0.537486771564\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:48:04 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=7500 train binary_f_1.000 <score>=0.736075133432\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:48:07 INFO 140684726773568] Iter[3] Batch [7000]#011Speed: 19433.95 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:48:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=7000 train binary_classification_accuracy <score>=0.733339380089\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:48:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=7000 train binary_classification_cross_entropy <loss>=0.534734374311\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:48:07 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=7000 train binary_f_1.000 <score>=0.738087732405\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:48:08 INFO 140667034941248] Iter[3] Batch [7000]#011Speed: 18962.77 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:48:08 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=7000 train binary_classification_accuracy <score>=0.733689901443\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:48:08 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=7000 train binary_classification_cross_entropy <loss>=0.534349377932\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:48:08 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=7000 train binary_f_1.000 <score>=0.738579081273\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:48:24 INFO 140658308478784] Iter[3] Batch [7500]#011Speed: 21606.35 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:48:24 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=7500 train binary_classification_accuracy <score>=0.732102919611\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:48:24 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=7500 train binary_classification_cross_entropy <loss>=0.536166035483\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:48:24 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=7500 train binary_f_1.000 <score>=0.737038711067\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:48:25 INFO 140009951377216] Iter[3] Batch [8000]#011Speed: 23603.87 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:48:25 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=8000 train binary_classification_accuracy <score>=0.731031746032\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:48:25 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=8000 train binary_classification_cross_entropy <loss>=0.537476871967\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:48:25 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=8000 train binary_f_1.000 <score>=0.736001782221\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:48:29 INFO 140684726773568] Iter[3] Batch [7500]#011Speed: 22730.96 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:48:29 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=7500 train binary_classification_accuracy <score>=0.733477536328\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:48:29 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=7500 train binary_classification_cross_entropy <loss>=0.534623751176\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:48:29 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=7500 train binary_f_1.000 <score>=0.73817176988\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:48:30 INFO 140667034941248] Iter[3] Batch [7500]#011Speed: 22245.79 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:48:30 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=7500 train binary_classification_accuracy <score>=0.73383522197\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:48:30 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=7500 train binary_classification_cross_entropy <loss>=0.534198542518\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:48:30 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=7500 train binary_f_1.000 <score>=0.738706272116\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:48:47 INFO 140658308478784] Iter[3] Batch [8000]#011Speed: 21374.21 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:48:47 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=8000 train binary_classification_accuracy <score>=0.732112485939\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:48:47 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=8000 train binary_classification_cross_entropy <loss>=0.536102367301\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:48:47 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=8000 train binary_f_1.000 <score>=0.737080581822\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:48:48 INFO 140009951377216] Iter[3] Batch [8500]#011Speed: 22289.52 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:48:48 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=8500 train binary_classification_accuracy <score>=0.730971885661\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:48:48 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=8500 train binary_classification_cross_entropy <loss>=0.537457330729\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:48:48 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=8500 train binary_f_1.000 <score>=0.735964367437\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:48:52 INFO 140684726773568] Iter[3] Batch [8000]#011Speed: 22618.49 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:48:52 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=8000 train binary_classification_accuracy <score>=0.733256217973\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:48:52 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=8000 train binary_classification_cross_entropy <loss>=0.534747392\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:48:52 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=8000 train binary_f_1.000 <score>=0.737975186293\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:48:53 INFO 140667034941248] Iter[3] Batch [8000]#011Speed: 22314.71 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:48:53 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=8000 train binary_classification_accuracy <score>=0.733674040745\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:48:53 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=8000 train binary_classification_cross_entropy <loss>=0.53426208408\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:48:53 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=8000 train binary_f_1.000 <score>=0.738593103651\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:49:10 INFO 140009951377216] Iter[3] Batch [9000]#011Speed: 22160.28 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:49:10 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=9000 train binary_classification_accuracy <score>=0.730922786357\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:49:10 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=9000 train binary_classification_cross_entropy <loss>=0.537439688318\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:49:10 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=9000 train binary_f_1.000 <score>=0.735950608293\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:49:12 INFO 140658308478784] Iter[3] Batch [8500]#011Speed: 20618.04 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:49:12 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=8500 train binary_classification_accuracy <score>=0.732021997412\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:49:12 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=8500 train binary_classification_cross_entropy <loss>=0.53612862411\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:49:12 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=8500 train binary_f_1.000 <score>=0.737004434631\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:49:14 INFO 140684726773568] Iter[3] Batch [8500]#011Speed: 22405.28 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:49:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=8500 train binary_classification_accuracy <score>=0.733323138454\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:49:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=8500 train binary_classification_cross_entropy <loss>=0.53468758713\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:49:14 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=8500 train binary_f_1.000 <score>=0.738086688729\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:49:15 INFO 140667034941248] Iter[3] Batch [8500]#011Speed: 22658.91 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:49:15 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=8500 train binary_classification_accuracy <score>=0.733775555817\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:49:15 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=8500 train binary_classification_cross_entropy <loss>=0.534180836481\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:49:15 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=8500 train binary_f_1.000 <score>=0.738761489917\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[31m[08/29/2018 13:49:34 INFO 140009951377216] Iter[3] Batch [9500]#011Speed: 21190.56 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:49:34 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=9500 train binary_classification_accuracy <score>=0.730824965793\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:49:34 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=9500 train binary_classification_cross_entropy <loss>=0.537443075164\u001b[0m\n",
      "\u001b[31m[08/29/2018 13:49:34 INFO 140009951377216] #quality_metric: host=algo-1, epoch=3, batch=9500 train binary_f_1.000 <score>=0.73589027945\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:49:35 INFO 140658308478784] Iter[3] Batch [9000]#011Speed: 21690.33 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:49:35 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=9000 train binary_classification_accuracy <score>=0.731989001222\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:49:35 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=9000 train binary_classification_cross_entropy <loss>=0.536130807328\u001b[0m\n",
      "\u001b[32m[08/29/2018 13:49:35 INFO 140658308478784] #quality_metric: host=algo-2, epoch=3, batch=9000 train binary_f_1.000 <score>=0.736950412773\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:49:37 INFO 140684726773568] Iter[3] Batch [9000]#011Speed: 21224.43 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:49:37 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=9000 train binary_classification_accuracy <score>=0.733209087879\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:49:37 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=9000 train binary_classification_cross_entropy <loss>=0.534734716135\u001b[0m\n",
      "\u001b[34m[08/29/2018 13:49:37 INFO 140684726773568] #quality_metric: host=algo-4, epoch=3, batch=9000 train binary_f_1.000 <score>=0.738006442852\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:49:38 INFO 140667034941248] Iter[3] Batch [9000]#011Speed: 21172.27 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:49:38 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=9000 train binary_classification_accuracy <score>=0.733682813021\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:49:38 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=9000 train binary_classification_cross_entropy <loss>=0.534197736401\u001b[0m\n",
      "\u001b[33m[08/29/2018 13:49:38 INFO 140667034941248] #quality_metric: host=algo-3, epoch=3, batch=9000 train binary_f_1.000 <score>=0.738627290099\u001b[0m\n",
      "\u001b[33m[08/29/2018 14:17:47 INFO 140667034941248] Iter[6] Batch [5000]#011Speed: 22940.79 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 14:17:47 INFO 140667034941248] #quality_metric: host=algo-3, epoch=6, batch=5000 train binary_classification_accuracy <score>=0.740372925415\u001b[0m\n",
      "\u001b[33m[08/29/2018 14:17:47 INFO 140667034941248] #quality_metric: host=algo-3, epoch=6, batch=5000 train binary_classification_cross_entropy <loss>=0.521227203253\u001b[0m\n",
      "\u001b[33m[08/29/2018 14:17:47 INFO 140667034941248] #quality_metric: host=algo-3, epoch=6, batch=5000 train binary_f_1.000 <score>=0.745380243635\u001b[0m\n",
      "\u001b[31m[08/29/2018 14:17:50 INFO 140009951377216] Iter[6] Batch [6000]#011Speed: 23049.73 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 14:17:50 INFO 140009951377216] #quality_metric: host=algo-1, epoch=6, batch=6000 train binary_classification_accuracy <score>=0.739215630728\u001b[0m\n",
      "\u001b[31m[08/29/2018 14:17:50 INFO 140009951377216] #quality_metric: host=algo-1, epoch=6, batch=6000 train binary_classification_cross_entropy <loss>=0.522563111186\u001b[0m\n",
      "\u001b[31m[08/29/2018 14:17:50 INFO 140009951377216] #quality_metric: host=algo-1, epoch=6, batch=6000 train binary_f_1.000 <score>=0.744274612699\u001b[0m\n",
      "\u001b[34m[08/29/2018 14:17:52 INFO 140684726773568] Iter[6] Batch [4500]#011Speed: 22178.60 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 14:17:52 INFO 140684726773568] #quality_metric: host=algo-4, epoch=6, batch=4500 train binary_classification_accuracy <score>=0.741176405243\u001b[0m\n",
      "\u001b[34m[08/29/2018 14:17:52 INFO 140684726773568] #quality_metric: host=algo-4, epoch=6, batch=4500 train binary_classification_cross_entropy <loss>=0.520156570811\u001b[0m\n",
      "\u001b[34m[08/29/2018 14:17:52 INFO 140684726773568] #quality_metric: host=algo-4, epoch=6, batch=4500 train binary_f_1.000 <score>=0.745963921432\u001b[0m\n",
      "\u001b[32m[08/29/2018 14:17:57 INFO 140658308478784] Iter[6] Batch [4000]#011Speed: 24624.29 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 14:17:57 INFO 140658308478784] #quality_metric: host=algo-2, epoch=6, batch=4000 train binary_classification_accuracy <score>=0.74248587853\u001b[0m\n",
      "\u001b[32m[08/29/2018 14:17:57 INFO 140658308478784] #quality_metric: host=algo-2, epoch=6, batch=4000 train binary_classification_cross_entropy <loss>=0.518721742347\u001b[0m\n",
      "\u001b[32m[08/29/2018 14:17:57 INFO 140658308478784] #quality_metric: host=algo-2, epoch=6, batch=4000 train binary_f_1.000 <score>=0.747187520857\u001b[0m\n",
      "\u001b[33m[08/29/2018 14:18:08 INFO 140667034941248] Iter[6] Batch [5500]#011Speed: 23622.21 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 14:18:08 INFO 140667034941248] #quality_metric: host=algo-3, epoch=6, batch=5500 train binary_classification_accuracy <score>=0.740415560807\u001b[0m\n",
      "\u001b[33m[08/29/2018 14:18:08 INFO 140667034941248] #quality_metric: host=algo-3, epoch=6, batch=5500 train binary_classification_cross_entropy <loss>=0.521139572235\u001b[0m\n",
      "\u001b[33m[08/29/2018 14:18:08 INFO 140667034941248] #quality_metric: host=algo-3, epoch=6, batch=5500 train binary_f_1.000 <score>=0.745414525471\u001b[0m\n",
      "\u001b[31m[08/29/2018 14:18:12 INFO 140009951377216] Iter[6] Batch [6500]#011Speed: 22237.51 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 14:18:12 INFO 140009951377216] #quality_metric: host=algo-1, epoch=6, batch=6500 train binary_classification_accuracy <score>=0.739298723273\u001b[0m\n",
      "\u001b[31m[08/29/2018 14:18:12 INFO 140009951377216] #quality_metric: host=algo-1, epoch=6, batch=6500 train binary_classification_cross_entropy <loss>=0.522484637415\u001b[0m\n",
      "\u001b[31m[08/29/2018 14:18:12 INFO 140009951377216] #quality_metric: host=algo-1, epoch=6, batch=6500 train binary_f_1.000 <score>=0.744378170068\u001b[0m\n",
      "\u001b[34m[08/29/2018 14:18:15 INFO 140684726773568] Iter[6] Batch [5000]#011Speed: 22059.58 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 14:18:15 INFO 140684726773568] #quality_metric: host=algo-4, epoch=6, batch=5000 train binary_classification_accuracy <score>=0.741306138772\u001b[0m\n",
      "\u001b[34m[08/29/2018 14:18:15 INFO 140684726773568] #quality_metric: host=algo-4, epoch=6, batch=5000 train binary_classification_cross_entropy <loss>=0.519986816693\u001b[0m\n",
      "\u001b[34m[08/29/2018 14:18:15 INFO 140684726773568] #quality_metric: host=algo-4, epoch=6, batch=5000 train binary_f_1.000 <score>=0.7461641937\u001b[0m\n",
      "\u001b[32m[08/29/2018 14:18:20 INFO 140658308478784] Iter[6] Batch [4500]#011Speed: 21690.46 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 14:18:20 INFO 140658308478784] #quality_metric: host=algo-2, epoch=6, batch=4500 train binary_classification_accuracy <score>=0.742430348811\u001b[0m\n",
      "\u001b[32m[08/29/2018 14:18:20 INFO 140658308478784] #quality_metric: host=algo-2, epoch=6, batch=4500 train binary_classification_cross_entropy <loss>=0.518713771763\u001b[0m\n",
      "\u001b[32m[08/29/2018 14:18:20 INFO 140658308478784] #quality_metric: host=algo-2, epoch=6, batch=4500 train binary_f_1.000 <score>=0.747161920523\u001b[0m\n",
      "\u001b[33m[08/29/2018 14:18:30 INFO 140667034941248] Iter[6] Batch [6000]#011Speed: 22880.96 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 14:18:30 INFO 140667034941248] #quality_metric: host=algo-3, epoch=6, batch=6000 train binary_classification_accuracy <score>=0.740405599067\u001b[0m\n",
      "\u001b[33m[08/29/2018 14:18:30 INFO 140667034941248] #quality_metric: host=algo-3, epoch=6, batch=6000 train binary_classification_cross_entropy <loss>=0.52111268139\u001b[0m\n",
      "\u001b[33m[08/29/2018 14:18:30 INFO 140667034941248] #quality_metric: host=algo-3, epoch=6, batch=6000 train binary_f_1.000 <score>=0.745475393315\u001b[0m\n",
      "\u001b[34m[08/29/2018 14:18:36 INFO 140684726773568] Iter[6] Batch [5500]#011Speed: 24549.49 samples/sec\u001b[0m\n",
      "\u001b[34m[08/29/2018 14:18:36 INFO 140684726773568] #quality_metric: host=algo-4, epoch=6, batch=5500 train binary_classification_accuracy <score>=0.741250863479\u001b[0m\n",
      "\u001b[34m[08/29/2018 14:18:36 INFO 140684726773568] #quality_metric: host=algo-4, epoch=6, batch=5500 train binary_classification_cross_entropy <loss>=0.520016276421\u001b[0m\n",
      "\u001b[34m[08/29/2018 14:18:36 INFO 140684726773568] #quality_metric: host=algo-4, epoch=6, batch=5500 train binary_f_1.000 <score>=0.746131317003\u001b[0m\n",
      "\u001b[31m[08/29/2018 14:18:35 INFO 140009951377216] Iter[6] Batch [7000]#011Speed: 22130.67 samples/sec\u001b[0m\n",
      "\u001b[31m[08/29/2018 14:18:35 INFO 140009951377216] #quality_metric: host=algo-1, epoch=6, batch=7000 train binary_classification_accuracy <score>=0.739243822311\u001b[0m\n",
      "\u001b[31m[08/29/2018 14:18:35 INFO 140009951377216] #quality_metric: host=algo-1, epoch=6, batch=7000 train binary_classification_cross_entropy <loss>=0.522498954791\u001b[0m\n",
      "\u001b[31m[08/29/2018 14:18:35 INFO 140009951377216] #quality_metric: host=algo-1, epoch=6, batch=7000 train binary_f_1.000 <score>=0.744338644434\u001b[0m\n",
      "\u001b[32m[08/29/2018 14:18:41 INFO 140658308478784] Iter[6] Batch [5000]#011Speed: 24235.05 samples/sec\u001b[0m\n",
      "\u001b[32m[08/29/2018 14:18:41 INFO 140658308478784] #quality_metric: host=algo-2, epoch=6, batch=5000 train binary_classification_accuracy <score>=0.742554689062\u001b[0m\n",
      "\u001b[32m[08/29/2018 14:18:41 INFO 140658308478784] #quality_metric: host=algo-2, epoch=6, batch=5000 train binary_classification_cross_entropy <loss>=0.5185684291\u001b[0m\n",
      "\u001b[32m[08/29/2018 14:18:41 INFO 140658308478784] #quality_metric: host=algo-2, epoch=6, batch=5000 train binary_f_1.000 <score>=0.747351127623\u001b[0m\n",
      "\u001b[33m[08/29/2018 14:18:50 INFO 140667034941248] Iter[6] Batch [6500]#011Speed: 24313.67 samples/sec\u001b[0m\n",
      "\u001b[33m[08/29/2018 14:18:50 INFO 140667034941248] #quality_metric: host=algo-3, epoch=6, batch=6500 train binary_classification_accuracy <score>=0.740457160437\u001b[0m\n",
      "\u001b[33m[08/29/2018 14:18:50 INFO 140667034941248] #quality_metric: host=algo-3, epoch=6, batch=6500 train binary_classification_cross_entropy <loss>=0.52105993359\u001b[0m\n",
      "\u001b[33m[08/29/2018 14:18:50 INFO 140667034941248] #quality_metric: host=algo-3, epoch=6, batch=6500 train binary_f_1.000 <score>=0.745522518846\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "fm = sagemaker.estimator.Estimator(container,\n",
    "                                   get_execution_role(), \n",
    "                                   train_instance_count=4, \n",
    "                                   train_instance_type='ml.c5.18xlarge',\n",
    "                                   output_path=output_prefix,\n",
    "                                   sagemaker_session=sagemaker.Session())\n",
    "\n",
    "fm.set_hyperparameters(feature_dim=nbFeatures,\n",
    "                      predictor_type='binary_classifier',\n",
    "                      mini_batch_size=1000,\n",
    "                      num_factors=64,\n",
    "                      epochs=100)\n",
    "\n",
    "fm.fit({'train': train_data, 'test': test_data})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Deploy model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "fm_predictor = fm.deploy(instance_type='ml.c4.xlarge', initial_instance_count=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fm_serializer(data):\n",
    "    js = {'instances': []}\n",
    "    for row in data:\n",
    "        js['instances'].append({'features': row.tolist()})\n",
    "    #print js\n",
    "    return json.dumps(js)\n",
    "\n",
    "fm_predictor.content_type = 'application/json'\n",
    "fm_predictor.serializer = fm_serializer\n",
    "fm_predictor.deserializer = json_deserializer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(X_test[1000:1010])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = fm_predictor.predict(X_test[2000:2010].toarray())\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print (Y_test[2000:2010])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(X_test[2000])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = fm_predictor.predict(X_test[2000:2001].toarray())\n",
    "print(result)\n",
    "print(Y_test[2000:2001])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python2",
   "language": "python",
   "name": "conda_python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.14"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}