Permalink
Find file Copy path
Fetching contributors…
Cannot retrieve contributors at this time
1 lines (1 sloc) 40.2 KB
{"paragraphs":[{"text":"%md\n### Note\n\nPlease view the [README](https://github.com/deeplearning4j/deeplearning4j/tree/master/dl4j-examples/tutorials/README.md) to learn about installing, setting up dependencies, and importing notebooks in Zeppelin","dateUpdated":"2018-06-25T08:48:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Note</h3>\n<p>Please view the <a href=\"https://github.com/deeplearning4j/deeplearning4j/tree/master/dl4j-examples/tutorials/README.md\">README</a> to learn about installing, setting up dependencies, and importing notebooks in Zeppelin</p>\n"}]},"apps":[],"jobName":"paragraph_1529916505084_93983843","id":"20180227-192621_547479636","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14848"},{"text":"%md\n\n### Background\n\nIn this tutorial we will use a neural network to forecast daily sea temperatures. The data consists of 2-dimensional temperature grids of 8 seas: Bengal, Korean, Black, Mediterranean, Arabian, Japan, Bohai, and Okhotsk Seas from 1981 to 2017. The raw data was taken from the Earth System Research Laboratory (https://www.esrl.noaa.gov/psd/) and preprocessed into CSV file. Each example consists of fifty 2-dimensional temperature grids, and every grid is represented by a single row in a CSV file. Thus, each sequence is represented by a CSV file with 50 rows.\n\nFor this task, we will use a convolutional LSTM neural network to forecast next-day sea temperatures for a given sequence of temperature grids. Recall, a convolutional network is most often used for image data like the MNIST dataset (dataset of handwritten images). A convolutional network is appropriate for this type of gridded data, since each point in the 2-dimensional grid is related to its neighbor points. Furthermore, the data is sequential, and each temperature grid is related to the previous grids. Because of these long and short term dependencies, a LSTM is fitting for this task too. For these two reasons, we will combine the aspects from these two different neural network architectures into a single convolutional LSTM network.\n\nFor more information on the convolutional LSTM network structure, see https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Ng_Beyond_Short_Snippets_2015_CVPR_paper.pdf\n\n","dateUpdated":"2018-06-25T08:48:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Background</h3>\n<p>In this tutorial we will use a neural network to forecast daily sea temperatures. The data consists of 2-dimensional temperature grids of 8 seas: Bengal, Korean, Black, Mediterranean, Arabian, Japan, Bohai, and Okhotsk Seas from 1981 to 2017. The raw data was taken from the Earth System Research Laboratory (https://www.esrl.noaa.gov/psd/) and preprocessed into CSV file. Each example consists of fifty 2-dimensional temperature grids, and every grid is represented by a single row in a CSV file. Thus, each sequence is represented by a CSV file with 50 rows.</p>\n<p>For this task, we will use a convolutional LSTM neural network to forecast next-day sea temperatures for a given sequence of temperature grids. Recall, a convolutional network is most often used for image data like the MNIST dataset (dataset of handwritten images). A convolutional network is appropriate for this type of gridded data, since each point in the 2-dimensional grid is related to its neighbor points. Furthermore, the data is sequential, and each temperature grid is related to the previous grids. Because of these long and short term dependencies, a LSTM is fitting for this task too. For these two reasons, we will combine the aspects from these two different neural network architectures into a single convolutional LSTM network.</p>\n<p>For more information on the convolutional LSTM network structure, see https://www.cv-foundation.org/openaccess/content_cvpr_2015/papers/Ng_Beyond_Short_Snippets_2015_CVPR_paper.pdf</p>\n"}]},"apps":[],"jobName":"paragraph_1529916505084_93983843","id":"20180227-192734_1160260969","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14849"},{"text":"%md\n### Imports","dateUpdated":"2018-06-25T08:48:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Imports</h3>\n"}]},"apps":[],"jobName":"paragraph_1529916505085_93599094","id":"20180227-193335_1287122015","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14850"},{"text":"import org.deeplearning4j.nn.api.OptimizationAlgorithm;\nimport org.deeplearning4j.nn.conf.NeuralNetConfiguration;\nimport org.deeplearning4j.nn.conf.layers.LSTM;\nimport org.deeplearning4j.nn.weights.WeightInit;\nimport org.nd4j.linalg.activations.Activation;\nimport org.nd4j.linalg.api.ndarray.INDArray;\nimport org.deeplearning4j.nn.multilayer.MultiLayerNetwork;\nimport org.nd4j.linalg.dataset.api.iterator.DataSetIterator;\nimport org.deeplearning4j.nn.conf.layers.RnnOutputLayer;\nimport org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator;\nimport org.nd4j.linalg.lossfunctions.LossFunctions.LossFunction;\nimport org.deeplearning4j.nn.conf.MultiLayerConfiguration;\nimport org.nd4j.linalg.dataset.DataSet;\nimport org.deeplearning4j.nn.conf.preprocessor.RnnToCnnPreProcessor;\nimport org.deeplearning4j.nn.conf.preprocessor.CnnToRnnPreProcessor;\nimport org.deeplearning4j.nn.conf.GradientNormalization;\nimport org.deeplearning4j.nn.conf.layers;\nimport org.deeplearning4j.eval.RegressionEvaluation;\nimport org.deeplearning4j.nn.conf.layers.ConvolutionLayer.Builder;\nimport org.deeplearning4j.nn.conf.layers.ConvolutionLayer;\nimport org.deeplearning4j.nn.conf.Updater;\n\nimport org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader;\nimport org.datavec.api.records.reader.SequenceRecordReader;\nimport org.datavec.api.split.NumberedFileInputSplit;\n\nimport java.io.File;\nimport java.net.URL;\nimport java.io.BufferedInputStream;\nimport java.io.FileInputStream;\nimport java.io.BufferedOutputStream;\nimport java.io.FileOutputStream;\n\nimport org.apache.commons.io.FilenameUtils;\nimport org.apache.commons.io.FileUtils;\nimport org.apache.commons.compress.archivers.tar.TarArchiveInputStream;\nimport org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream;\nimport org.apache.commons.compress.archivers.tar.TarArchiveEntry;\n\n","dateUpdated":"2018-06-25T08:48:25+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","results":{},"enabled":true,"editorSetting":{"language":"scala"}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"import org.deeplearning4j.nn.api.OptimizationAlgorithm\nimport org.deeplearning4j.nn.conf.NeuralNetConfiguration\nimport org.deeplearning4j.nn.conf.layers.GravesLSTM\nimport org.deeplearning4j.nn.weights.WeightInit\nimport org.nd4j.linalg.activations.Activation\nimport org.nd4j.linalg.api.ndarray.INDArray\nimport org.deeplearning4j.nn.multilayer.MultiLayerNetwork\nimport org.nd4j.linalg.dataset.api.iterator.DataSetIterator\nimport org.deeplearning4j.nn.conf.layers.RnnOutputLayer\nimport org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator\nimport org.nd4j.linalg.lossfunctions.LossFunctions.LossFunction\nimport org.deeplearning4j.nn.conf.MultiLayerConfiguration\nimport org.nd4j.linalg.dataset.DataSet\nimport org.deeplearning4j.nn.conf.preprocessor.RnnToCnnPreProcessor\nimport org.deeplearning4j.nn.conf.preprocessor.CnnToRnnPreProcessor\nimport org.deeplearning4j.nn.conf.GradientNormalization\nimport org.deeplearning4j.nn.conf.layers\nimport org.deeplearning4j.eval.RegressionEvaluation\nimport org.deeplearning4j.nn.conf.layers.ConvolutionLayer.Builder\nimport org.deeplearning4j.nn.conf.layers.ConvolutionLayer\nimport org.deeplearning4j.nn.conf.Updater\nimport org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader\nimport org.datavec.api.records.reader.SequenceRecordReader\nimport org.datavec.api.split.NumberedFileInputSplit\nimport java.io.File\nimport java.net.URL\nimport java.io.BufferedInputStream\nimport java.io.FileInputStream\nimport java.io.BufferedOutputStream\nimport java.io.FileOutputStream\nimport org.apache.commons.io.FilenameUtils\nimport org.apache.commons.io.FileUtils\nimport org.apache.commons.compress.archivers.tar.TarArchiveInputStream\nimport org.apache.commons.compress.compressors.gzip.GzipCompressorInputStream\nimport org.apache.commons.compress.archivers.tar.TarArchiveEntry\n"}]},"apps":[],"jobName":"paragraph_1529916505085_93599094","id":"20180226-040931_717744155","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14851"},{"text":"%md\n### Download Data","dateUpdated":"2018-06-25T08:48:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Download Data</h3>\n"}]},"apps":[],"jobName":"paragraph_1529916505085_93599094","id":"20180227-193345_527268888","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14852"},{"text":"%md\nTo download the data, we will create a temporary directory that will store the data files, extract the tar.gz file from the url, and place it in the specified directory.","dateUpdated":"2018-06-25T08:48:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>To download the data, we will create a temporary directory that will store the data files, extract the tar.gz file from the url, and place it in the specified directory.</p>\n"}]},"apps":[],"jobName":"paragraph_1529916505085_93599094","id":"20180228-204517_252559836","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14853"},{"text":"val DATA_URL = \"https://bpstore1.blob.core.windows.net/seatemp/sea_temp.tar.gz\"\nval DATA_PATH = FilenameUtils.concat(System.getProperty(\"java.io.tmpdir\"), \"dl4j_seas/\")","dateUpdated":"2018-06-25T08:48:25+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","results":{},"enabled":true,"editorSetting":{"language":"scala"}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"DATA_URL: String = https://bpstore1.blob.core.windows.net/seatemp/sea_temp.tar.gz\nDATA_PATH: String = /tmp/dl4j_seas/\n"}]},"apps":[],"jobName":"paragraph_1529916505086_94753341","id":"20180226-045411_2004973787","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14854"},{"text":"val directory = new File(DATA_PATH)\ndirectory.mkdir() \n\nval archizePath = DATA_PATH + \"sea_temp.tar.gz\"\nval archiveFile = new File(archizePath)\nval extractedPath = DATA_PATH + \"sea_temp\" \nval extractedFile = new File(extractedPath)\n\nFileUtils.copyURLToFile(new URL(DATA_URL), archiveFile) ","dateUpdated":"2018-06-25T08:48:25+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","results":{},"enabled":true,"editorSetting":{"language":"scala"}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"directory: java.io.File = /tmp/dl4j_seas\nres3: Boolean = false\narchizePath: String = /tmp/dl4j_seas/sea_temp.tar.gz\narchiveFile: java.io.File = /tmp/dl4j_seas/sea_temp.tar.gz\nextractedPath: String = /tmp/dl4j_seas/sea_temp\nextractedFile: java.io.File = /tmp/dl4j_seas/sea_temp\n"}]},"apps":[],"jobName":"paragraph_1529916505086_94753341","id":"20180226-045622_1883168318","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14855"},{"text":"%md\nWe will then extract the data from the tar.gz file, recreate directories within the tar.gz file into our temporary directories, and copy the files from the tar.gz file.","dateUpdated":"2018-06-25T08:48:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>We will then extract the data from the tar.gz file, recreate directories within the tar.gz file into our temporary directories, and copy the files from the tar.gz file.</p>\n"}]},"apps":[],"jobName":"paragraph_1529916505086_94753341","id":"20180228-204547_621784709","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14856"},{"text":"var fileCount = 0\nvar dirCount = 0\nval BUFFER_SIZE = 4096\nval tais = new TarArchiveInputStream(new GzipCompressorInputStream( new BufferedInputStream( new FileInputStream(archizePath))))\n\nvar entry = tais.getNextEntry().asInstanceOf[TarArchiveEntry]\n\nwhile(entry != null){\n if (entry.isDirectory()) {\n new File(DATA_PATH + entry.getName()).mkdirs()\n dirCount = dirCount + 1\n fileCount = 0\n }\n else {\n \n val data = new Array[scala.Byte](4 * BUFFER_SIZE)\n\n val fos = new FileOutputStream(DATA_PATH + entry.getName());\n val dest = new BufferedOutputStream(fos, BUFFER_SIZE);\n var count = tais.read(data, 0, BUFFER_SIZE)\n \n while (count != -1) {\n dest.write(data, 0, count)\n count = tais.read(data, 0, BUFFER_SIZE)\n }\n \n dest.close()\n fileCount = fileCount + 1\n }\n if(fileCount % 1000 == 0){\n print(\".\")\n }\n \n entry = tais.getNextEntry().asInstanceOf[TarArchiveEntry]\n}\n\n\n\n","dateUpdated":"2018-06-25T08:48:25+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","results":{},"enabled":true,"editorSetting":{"language":"scala"}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"fileCount: Int = 0\ndirCount: Int = 0\nBUFFER_SIZE: Int = 4096\ntais: org.apache.commons.compress.archivers.tar.TarArchiveInputStream = org.apache.commons.compress.archivers.tar.TarArchiveInputStream@2257b9ce\nentry: org.apache.commons.compress.archivers.tar.TarArchiveEntry = org.apache.commons.compress.archivers.tar.TarArchiveEntry@4a7f9eeb\n......."}]},"apps":[],"jobName":"paragraph_1529916505086_94753341","id":"20180226-045639_615846226","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14857"},{"text":"%md\n### DataSetIterators","dateUpdated":"2018-06-25T08:48:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>DataSetIterators</h3>\n"}]},"apps":[],"jobName":"paragraph_1529916505086_94753341","id":"20180228-204640_1112094027","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14858"},{"text":"%md\nNext we will convert the raw data (csv files) into DataSetIterators, which will be fed into a neural network. Our training data will have 1700 examples which will be represented by a single DataSetIterator, and the testing data will have 404 examples which will be represented by a separate DataSet Iterator.","dateUpdated":"2018-06-25T08:48:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>Next we will convert the raw data (csv files) into DataSetIterators, which will be fed into a neural network. Our training data will have 1700 examples which will be represented by a single DataSetIterator, and the testing data will have 404 examples which will be represented by a separate DataSet Iterator.</p>\n"}]},"apps":[],"jobName":"paragraph_1529916505087_94368592","id":"20180228-204709_1807307832","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14859"},{"text":"val path = FilenameUtils.concat(DATA_PATH, \"sea_temp/\") // set parent directory\n\nval featureBaseDir = FilenameUtils.concat(path, \"features\") // set feature directory\nval targetsBaseDir = FilenameUtils.concat(path, \"targets\") // set label directory","dateUpdated":"2018-06-25T08:48:25+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","results":{},"enabled":true,"editorSetting":{"language":"scala"}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"path: String = /tmp/dl4j_seas/sea_temp/\nfeatureBaseDir: String = /tmp/dl4j_seas/sea_temp/features\ntargetsBaseDir: String = /tmp/dl4j_seas/sea_temp/targets\n"}]},"apps":[],"jobName":"paragraph_1529916505087_94368592","id":"20180228-202742_352342907","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14860"},{"text":"%md\nWe first initialize CSVSequenceRecordReaders, which will parse the raw data into record-like format. Then the SequenceRecordReaderDataSetIterators can be created using the RecordReaders. Since each example has exaclty 50 timesteps, an alignment mode of equal length is needed. Note also that this is a regression-based task and not a classification one.","dateUpdated":"2018-06-25T08:48:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>We first initialize CSVSequenceRecordReaders, which will parse the raw data into record-like format. Then the SequenceRecordReaderDataSetIterators can be created using the RecordReaders. Since each example has exaclty 50 timesteps, an alignment mode of equal length is needed. Note also that this is a regression-based task and not a classification one.</p>\n"}]},"apps":[],"jobName":"paragraph_1529916505087_94368592","id":"20180228-204701_1601580223","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14861"},{"text":"val numSkipLines = 1;\nval regression = true;\nval batchSize = 32;\n\nval trainFeatures = new CSVSequenceRecordReader(numSkipLines, \",\");\ntrainFeatures.initialize( new NumberedFileInputSplit(featureBaseDir + \"/%d.csv\", 1, 1936));\nval trainTargets = new CSVSequenceRecordReader(numSkipLines, \",\");\ntrainTargets.initialize(new NumberedFileInputSplit(targetsBaseDir + \"/%d.csv\", 1, 1936));\n\nval train = new SequenceRecordReaderDataSetIterator(trainFeatures, trainTargets, batchSize,\n 10, regression, SequenceRecordReaderDataSetIterator.AlignmentMode.EQUAL_LENGTH);\n \n \nval testFeatures = new CSVSequenceRecordReader(numSkipLines, \",\");\ntestFeatures.initialize( new NumberedFileInputSplit(featureBaseDir + \"/%d.csv\", 1937, 2089));\nval testTargets = new CSVSequenceRecordReader(numSkipLines, \",\");\ntestTargets.initialize(new NumberedFileInputSplit(targetsBaseDir + \"/%d.csv\", 1937, 2089));\n\nval test = new SequenceRecordReaderDataSetIterator(testFeatures, testTargets, batchSize,\n 10, regression, SequenceRecordReaderDataSetIterator.AlignmentMode.EQUAL_LENGTH);","dateUpdated":"2018-06-25T08:48:25+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","results":{},"enabled":true,"editorSetting":{"language":"scala"}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"numSkipLines: Int = 1\nregression: Boolean = true\nbatchSize: Int = 32\ntrainFeatures: org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader = org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader@9032876\ntrainTargets: org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader = org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader@86324e7\ntrain: org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator = org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator@1e2e7948\ntestFeatures: org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader = org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader@1b6d187d\ntestTargets: org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader = org.datavec.api.records.reader.impl.csv.CSVSequenceRecordReader@10d34f3d\ntest: org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator = org.deeplearning4j.datasets.datavec.SequenceRecordReaderDataSetIterator@7346b46d\n"}]},"apps":[],"jobName":"paragraph_1529916505087_94368592","id":"20180228-203010_961096132","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14862"},{"text":"%md \n\n### Neural Network","dateUpdated":"2018-06-25T08:48:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Neural Network</h3>\n"}]},"apps":[],"jobName":"paragraph_1529916505088_-215354273","id":"20180227-192558_1730077226","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14863"},{"text":"%md\nThe next task is to initialize the parameters for the convolutional LSTM neural network and then set up the neural network configuration.","dateUpdated":"2018-06-25T08:48:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>The next task is to initialize the parameters for the convolutional LSTM neural network and then set up the neural network configuration.</p>\n"}]},"apps":[],"jobName":"paragraph_1529916505088_-215354273","id":"20180228-205115_481934288","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14864"},{"text":"val V_HEIGHT = 13;\nval V_WIDTH = 4;\nval kernelSize = 2;\nval numChannels = 1;","dateUpdated":"2018-06-25T08:48:25+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","results":{},"enabled":true,"editorSetting":{"language":"scala"}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"V_HEIGHT: Int = 13\nV_WIDTH: Int = 4\nkernelSize: Int = 2\nnumChannels: Int = 1\n"}]},"apps":[],"jobName":"paragraph_1529916505088_-215354273","id":"20180226-040942_1672801042","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14865"},{"text":"%md\n\nIn the neural network configuraiton we will use the convolutional layer, LSTM layer, and output layer in success. In order to do this, we need to use the RnnToCnnPreProcessor and CnnToRnnPreprocessor. The RnnToCnnPreProcessor is used to reshape the 3-dimensional input from [batch size, height x width of grid, time series length ] into a 4 dimensional shape [number of examples x time series length , channels, width, height] which is suitable as input to a convolutional layer. The CnnToRnnPreProcessor is then used in a later layer to convert this convolutional shape back to the original 3-dimensional shape.","dateUpdated":"2018-06-25T08:48:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>In the neural network configuraiton we will use the convolutional layer, LSTM layer, and output layer in success. In order to do this, we need to use the RnnToCnnPreProcessor and CnnToRnnPreprocessor. The RnnToCnnPreProcessor is used to reshape the 3-dimensional input from [batch size, height x width of grid, time series length ] into a 4 dimensional shape [number of examples x time series length , channels, width, height] which is suitable as input to a convolutional layer. The CnnToRnnPreProcessor is then used in a later layer to convert this convolutional shape back to the original 3-dimensional shape.</p>\n"}]},"apps":[],"jobName":"paragraph_1529916505088_-215354273","id":"20180227-194623_1042566542","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14866"},{"text":"val conf = new NeuralNetConfiguration.Builder()\n .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT)\n .seed(12345)\n .weightInit(WeightInit.XAVIER)\n .list()\n .layer(0, new ConvolutionLayer.Builder(kernelSize, kernelSize)\n .updater(Updater.ADAGRAD)\n //.learningRate(0.005)\n .nIn(1) //1 channel\n .nOut(7)\n .stride(2, 2)\n .activation(Activation.RELU)\n .build())\n .layer(1, new LSTM.Builder()\n .activation(Activation.SOFTSIGN)\n .nIn(84)\n .nOut(200)\n .updater(Updater.ADAGRAD)\n //.learningRate(0.0005)\n .gradientNormalization(GradientNormalization.ClipElementWiseAbsoluteValue)\n .gradientNormalizationThreshold(10)\n .build())\n .layer(2, new RnnOutputLayer.Builder(LossFunction.MSE)\n .activation(Activation.IDENTITY)\n .nIn(200)\n .updater(Updater.ADAGRAD)\n //.learningRate(0.0005)\n .nOut(52)\n .gradientNormalization(GradientNormalization.ClipElementWiseAbsoluteValue)\n .gradientNormalizationThreshold(10)\n .build())\n .inputPreProcessor(0, new RnnToCnnPreProcessor(V_HEIGHT, V_WIDTH, numChannels))\n .inputPreProcessor(1, new CnnToRnnPreProcessor(6, 2, 7 ))\n .pretrain(false).backprop(true)\n .build();\n \nval net = new MultiLayerNetwork(conf);\nnet.init();","dateUpdated":"2018-06-25T08:48:25+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","results":{},"enabled":true,"editorSetting":{"language":"scala"}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"warning: there were 3 deprecation warning(s); re-run with -deprecation for details\nconf: org.deeplearning4j.nn.conf.MultiLayerConfiguration = \n{\n \"backprop\" : true,\n \"backpropType\" : \"Standard\",\n \"cacheMode\" : \"NONE\",\n \"confs\" : [ {\n \"cacheMode\" : \"NONE\",\n \"epochCount\" : 0,\n \"iterationCount\" : 0,\n \"l1ByParam\" : { },\n \"l2ByParam\" : { },\n \"layer\" : {\n \"convolution\" : {\n \"activationFn\" : {\n \"ReLU\" : { }\n },\n \"biasInit\" : 0.0,\n \"biasUpdater\" : null,\n \"constraints\" : null,\n \"convolutionMode\" : \"Truncate\",\n \"cudnnAlgoMode\" : \"PREFER_FASTEST\",\n \"cudnnBwdDataAlgo\" : null,\n \"cudnnBwdFilterAlgo\" : null,\n \"cudnnFwdAlgo\" : null,\n \"dilation\" : [ 1, 1 ],\n \"dist\" : null,\n \"gradientNormalization\" : \"None\",\n \"gradientNormalizationThreshold\" : 1.0,\n ...net: org.deeplearning4j.nn.multilayer.MultiLayerNetwork = org.deeplearning4j.nn.multilayer.MultiLayerNetwork@5cc97c1\n"}]},"apps":[],"jobName":"paragraph_1529916505089_-215739021","id":"20180226-041427_1422510576","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14867"},{"text":"%md\n\n### Model Training","dateUpdated":"2018-06-25T08:48:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Model Training</h3>\n"}]},"apps":[],"jobName":"paragraph_1529916505089_-215739021","id":"20180227-193516_850320192","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14868"},{"text":"%md\nTo train the model, we use 25 epochs with a for loop and simply call the fit method of the MultiLayerNetwork.","dateUpdated":"2018-06-25T08:48:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>To train the model, we use 25 epochs with a for loop and simply call the fit method of the MultiLayerNetwork.</p>\n"}]},"apps":[],"jobName":"paragraph_1529916505089_-215739021","id":"20180228-205146_179808891","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14869"},{"text":"// Train model on training set\n\nfor( epoch <- 1 to 25){\n println(\"Epoch \"+ epoch);\n net.fit( train );\n train.reset();\n}","dateUpdated":"2018-06-25T08:48:25+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","results":{},"enabled":true,"editorSetting":{"language":"scala"}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"Epoch 1\nEpoch 2\nEpoch 3\nEpoch 4\nEpoch 5\nEpoch 6\nEpoch 7\nEpoch 8\nEpoch 9\nEpoch 10\nEpoch 11\nEpoch 12\nEpoch 13\nEpoch 14\nEpoch 15\nEpoch 16\nEpoch 17\nEpoch 18\nEpoch 19\nEpoch 20\nEpoch 21\nEpoch 22\nEpoch 23\nEpoch 24\nEpoch 25\n"}]},"apps":[],"jobName":"paragraph_1529916505089_-215739021","id":"20180226-041443_1106893390","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14870"},{"text":"%md\n### Model Evaluation","dateUpdated":"2018-06-25T08:48:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<h3>Model Evaluation</h3>\n"}]},"apps":[],"jobName":"paragraph_1529916505090_-214584775","id":"20180227-193633_1813883250","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14871"},{"text":"%md\nWe will now evaluate our trained model. Note that we will use RegressionEvaluation, since our task is a regression and not a classification task.","dateUpdated":"2018-06-25T08:48:25+0000","config":{"tableHide":false,"editorSetting":{"language":"markdown","editOnDblClick":true},"colWidth":12,"editorMode":"ace/mode/markdown","editorHide":true,"results":{},"enabled":true},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"HTML","data":"<p>We will now evaluate our trained model. Note that we will use RegressionEvaluation, since our task is a regression and not a classification task.</p>\n"}]},"apps":[],"jobName":"paragraph_1529916505090_-214584775","id":"20180228-205218_1696106790","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14872"},{"text":"val eval = net.evaluateRegression(test);\n\ntest.reset();\nprintln()\n\nprintln( eval.stats() );","dateUpdated":"2018-06-25T08:48:25+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","results":{},"enabled":true,"editorSetting":{"language":"scala"}},"settings":{"params":{},"forms":{}},"results":{"code":"SUCCESS","msg":[{"type":"TEXT","data":"eval: org.deeplearning4j.eval.RegressionEvaluation = RegressionEvaluation(initialized=true, columnNames=[col_0, col_1, col_2, col_3, col_4, col_5, col_6, col_7, col_8, col_9, col_10, col_11, col_12, col_13, col_14, col_15, col_16, col_17, col_18, col_19, col_20, col_21, col_22, col_23, col_24, col_25, col_26, col_27, col_28, col_29, col_30, col_31, col_32, col_33, col_34, col_35, col_36, col_37, col_38, col_39, col_40, col_41, col_42, col_43, col_44, col_45, col_46, col_47, col_48, col_49, col_50, col_51], precision=5, exampleCountPerColumn=[7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.00, 7,650.0...\nColumn MSE MAE RMSE RSE PC R^2 \ncol_0 1.85482e+00 8.18130e-01 1.36192e+00 1.95191e-02 9.90645e-01 9.79914e-01 \ncol_1 1.71444e+00 7.62419e-01 1.30936e+00 1.80415e-02 9.91360e-01 9.81429e-01 \ncol_2 1.73293e+00 7.71783e-01 1.31641e+00 1.82226e-02 9.91201e-01 9.81236e-01 \ncol_3 1.80103e+00 7.87673e-01 1.34202e+00 1.88237e-02 9.90923e-01 9.80610e-01 \ncol_4 1.75717e+00 7.84176e-01 1.32558e+00 1.85604e-02 9.91159e-01 9.80897e-01 \ncol_5 1.62221e+00 7.28274e-01 1.27366e+00 1.71386e-02 9.91864e-01 9.82352e-01 \ncol_6 1.53927e+00 6.84305e-01 1.24067e+00 1.62365e-02 9.92364e-01 9.83274e-01 \ncol_7 1.62093e+00 7.14761e-01 1.27316e+00 1.69972e-02 9.91938e-01 9.82486e-01 \ncol_8 1.68882e+00 7.54865e-01 1.29954e+00 1.79221e-02 9.91498e-01 9.81554e-01 \ncol_9 1.53570e+00 6.89777e-01 1.23924e+00 1.63035e-02 9.92355e-01 9.83213e-01 \ncol_10 1.53987e+00 6.78374e-01 1.24091e+00 1.63174e-02 9.92362e-01 9.83189e-01 \ncol_11 1.57873e+00 7.14330e-01 1.25647e+00 1.66301e-02 9.92178e-01 9.82862e-01 \ncol_12 1.58736e+00 7.19045e-01 1.25991e+00 1.69223e-02 9.92078e-01 9.82585e-01 \ncol_13 1.53868e+00 6.79158e-01 1.24044e+00 1.64154e-02 9.92332e-01 9.83098e-01 \ncol_14 1.52813e+00 6.75412e-01 1.23617e+00 1.62788e-02 9.92467e-01 9.83232e-01 \ncol_15 1.63093e+00 7.24261e-01 1.27708e+00 1.72697e-02 9.91964e-01 9.82206e-01 \ncol_16 1.56559e+00 7.08976e-01 1.25123e+00 1.67690e-02 9.92179e-01 9.82746e-01 \ncol_17 1.45944e+00 6.50576e-01 1.20807e+00 1.56633e-02 9.92658e-01 9.83874e-01 \ncol_18 1.55862e+00 6.87859e-01 1.24845e+00 1.67064e-02 9.92205e-01 9.82793e-01 \ncol_19 1.57934e+00 7.02607e-01 1.25672e+00 1.68333e-02 9.92121e-01 9.82660e-01 \ncol_20 1.52430e+00 6.78341e-01 1.23463e+00 1.64166e-02 9.92197e-01 9.83110e-01 \ncol_21 1.47002e+00 6.56704e-01 1.21244e+00 1.58590e-02 9.92559e-01 9.83673e-01 \ncol_22 1.57344e+00 6.96654e-01 1.25437e+00 1.69532e-02 9.92111e-01 9.82542e-01 \ncol_23 1.61090e+00 7.21663e-01 1.26921e+00 1.72664e-02 9.91891e-01 9.82217e-01 \ncol_24 1.56159e+00 6.91928e-01 1.24964e+00 1.69036e-02 9.91997e-01 9.82611e-01 \ncol_25 1.46744e+00 6.51722e-01 1.21138e+00 1.58874e-02 9.92518e-01 9.83650e-01 \ncol_26 1.49904e+00 6.66568e-01 1.22435e+00 1.61994e-02 9.92405e-01 9.83323e-01 \ncol_27 1.61818e+00 7.11384e-01 1.27208e+00 1.74135e-02 9.91849e-01 9.82072e-01 \ncol_28 1.53726e+00 6.83343e-01 1.23986e+00 1.66940e-02 9.92114e-01 9.82829e-01 \ncol_29 1.49821e+00 6.69927e-01 1.22401e+00 1.62511e-02 9.92380e-01 9.83282e-01 \ncol_30 1.52075e+00 6.85024e-01 1.23319e+00 1.64528e-02 9.92317e-01 9.83072e-01 \ncol_31 1.55868e+00 6.94010e-01 1.24847e+00 1.68008e-02 9.92202e-01 9.82711e-01 \ncol_32 1.53357e+00 6.84129e-01 1.23837e+00 1.66974e-02 9.92069e-01 9.82829e-01 \ncol_33 1.51121e+00 6.73379e-01 1.22931e+00 1.64274e-02 9.92303e-01 9.83108e-01 \ncol_34 1.52704e+00 6.93791e-01 1.23574e+00 1.65544e-02 9.92308e-01 9.82977e-01 \ncol_35 1.54869e+00 6.97562e-01 1.24447e+00 1.67295e-02 9.92238e-01 9.82797e-01 \ncol_36 1.56622e+00 6.80601e-01 1.25149e+00 1.71295e-02 9.91786e-01 9.82387e-01 \ncol_37 1.50162e+00 6.65254e-01 1.22541e+00 1.64140e-02 9.92358e-01 9.83128e-01 \ncol_38 1.51583e+00 6.67052e-01 1.23119e+00 1.65427e-02 9.92235e-01 9.83000e-01 \ncol_39 1.63063e+00 7.20754e-01 1.27696e+00 1.77261e-02 9.91880e-01 9.81786e-01 \ncol_40 1.51194e+00 6.73928e-01 1.22961e+00 1.66450e-02 9.92156e-01 9.82890e-01 \ncol_41 1.61124e+00 7.03437e-01 1.26935e+00 1.77709e-02 9.91737e-01 9.81739e-01 \ncol_42 1.61306e+00 7.12124e-01 1.27006e+00 1.77872e-02 9.91776e-01 9.81730e-01 \ncol_43 1.77267e+00 7.65805e-01 1.33142e+00 1.94563e-02 9.91121e-01 9.80023e-01 \ncol_44 1.67468e+00 7.34147e-01 1.29409e+00 1.85837e-02 9.91259e-01 9.80904e-01 \ncol_45 1.73448e+00 7.46950e-01 1.31700e+00 1.93163e-02 9.90953e-01 9.80161e-01 \ncol_46 1.84811e+00 7.74505e-01 1.35945e+00 2.05801e-02 9.90488e-01 9.78876e-01 \ncol_47 2.06485e+00 8.43610e-01 1.43696e+00 2.28423e-02 9.89558e-01 9.76564e-01 \ncol_48 1.76594e+00 7.71466e-01 1.32889e+00 1.97377e-02 9.90755e-01 9.79727e-01 \ncol_49 1.97987e+00 8.24289e-01 1.40708e+00 2.22171e-02 9.89591e-01 9.77192e-01 \ncol_50 2.09795e+00 8.43424e-01 1.44843e+00 2.35141e-02 9.89143e-01 9.75874e-01 \ncol_51 2.25343e+00 8.86249e-01 1.50114e+00 2.50340e-02 9.88568e-01 9.74325e-01 \n\n"}]},"apps":[],"jobName":"paragraph_1529916505090_-214584775","id":"20180227-193643_1887676521","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14873"},{"dateUpdated":"2018-06-25T08:48:25+0000","config":{"colWidth":12,"editorMode":"ace/mode/scala","results":{},"enabled":true,"editorSetting":{"language":"scala","editOnDblClick":false}},"settings":{"params":{},"forms":{}},"apps":[],"jobName":"paragraph_1529916505090_-214584775","id":"20180228-203637_482752207","dateCreated":"2018-06-25T08:48:25+0000","status":"READY","errorMessage":"","progressUpdateIntervalMs":500,"$$hashKey":"object:14874"}],"name":"Sea Temperature Convolutional LSTM Example","id":"2DKGFM8FZ","angularObjects":{"2DKTVHEQG:existing_process":[],"2DJ4SFCPD:existing_process":[],"2DJJJ8C1V:existing_process":[],"2DHBWPF6M:existing_process":[],"2DJB51UJ1:existing_process":[],"2DHQZP5Q3:existing_process":[],"2DKS7J9U9:existing_process":[],"2DHPXD7E1:existing_process":[]},"config":{"looknfeel":"default","personalizedMode":"false"},"info":{}}