WIP Neuraxle refactor #16

alexbrillant · 2020-01-08T19:00:17Z

Use Neuraxle TensorflowV1ModelStep
Use Dynamic RNN to predict sequence of dynamic lengths
Clean tensorflow code & placeholders

TODO: comments & docstrings...
Note : it wouldn't take long to migrate this version to tensorflow 2 !

How to use TrainingHelper and InferenceHelper together :

https://stackoverflow.com/questions/49134432/how-to-use-tensorflow-seq2seq-without-embeddings

model.py

…s infinitely decoding

guillaume-chevalier · 2020-01-13T01:54:11Z

Here is what I think we could do with TF2:

last_encoder_state, encoder_outputs = apply_encoder(encoder, tf_data_inputs)
last_encoder_output = encoder_outputs[:, -1, :]

stacked_cells = [gru_cell(...) for _ in range(stacked)]

rnn = RNN(stacked_cells)

decoder_outputs = []
state = last_encoder_state
for time_step in range(self.output_seq_len): 
    state, out = rnn(state_cell=state, input=last_encoder_output)  # repeat last encoder output as an input (cheap but very okay to do this here).
    decoder_outputs.append(out)

return decoder_outputs

This is some dirty pseudocode I've just done. For instance, the apply_encoder method is extracted in my code above as it seems not to be the problem we focus on. I think it would be possible to extract method as such for the decoder.

guillaume-chevalier

First code review pass

guillaume-chevalier · 2020-01-13T01:24:04Z

model.py

+        name='data_inputs'
+    )
+
+    # shape: (batch_size, seq_length, input_dim)


Suggested change

# shape: (batch_size, seq_length, input_dim)

# shape: (batch_size, seq_length, output_dim)

guillaume-chevalier · 2020-01-13T01:25:45Z

model.py

+    # shape: (batch_size)
+    target_sequence_length = tf.placeholder(dtype=tf.int32, shape=[None], name='expected_outputs_length')


Wrong dimension comment on an unused variable. Delete those 2 lines?

guillaume-chevalier · 2020-01-13T01:27:21Z

model.py

+    return decoder_outputs_training, decoder_outputs_inference
+
+
+def create_encoder(step, data_inputs):


Dimension comments of inputs and outputs will be important in those small functions

guillaume-chevalier · 2020-01-13T01:30:29Z

model.py

+
+
+def create_inference_decoder(step: TensorflowV1ModelStep, encoder_state, decoder_cell):
+    start_inputs = tf.constant(GO_TOKEN, shape=[step.hyperparams['batch_size'], step.hyperparams['output_dim']])


One strange thing I noticed here: this "start_inputs" is 2D here, whereas it's 3D in the create_training_decoder. Might be something to look at.

But I'd delete the create_inference_decoder function anyway, and rather instead use the other one for both test and train.

guillaume-chevalier · 2020-01-13T01:33:05Z

model.py

+
+def create_training_decoder(step: TensorflowV1ModelStep, encoder_state, decoder_cell):
+    go_tokens = tf.constant(GO_TOKEN, shape=[step.hyperparams['batch_size'], 1, step.hyperparams['output_dim']])
+    inputs = tf.concat([go_tokens, step['expected_outputs']], axis=1)


The error might be here: I would use Go tokens everywhere. OR even better: I would use the encoder_outputs at the last time step, which is probably encoder_outputs[:, -1, :], as a go token everywhere.

This way, we would have a decoder that is the same at inference (prediction) time AND at training time. We only need one function here. That might be the problem you were facing, explaining why the things didn't work.

Another thing that could explain why the things doesn't work is that you may have not shared parameters between the two decoders, despite it looks like you've properly shared them with the method create_stacked_rnn. So perhaps that you have two decoders and one encoder by mistake (although I think you're okay).

guillaume-chevalier · 2020-01-13T01:33:52Z

model.py

+
+    decoder_cell = create_stacked_rnn(step)
+    decoder_outputs_training = create_training_decoder(step, encoder_state, decoder_cell)
+    decoder_outputs_inference = create_inference_decoder(step, encoder_state, decoder_cell)


Let's keep only one decoder here. See other comments.

To be more precise, I would edit something in the training decoder and delete the inference one that has the InferenceHelper for instance (that's too complicated to explain in a short training for people to use it and what I suggest will work fine).

guillaume-chevalier · 2020-01-13T01:43:51Z

model.py

+    return output
+
+
+def create_decoder_outputs(step, helper, encoder_state, decoder_cell):


I didn't review this function, but it looks overly complicated.

Could we base ourselves as much as possible on my original seq2seq example here without many changes? (if that is possible anyhow)

Or see the TF2 example I've commented above this PR's review.

guillaume-chevalier · 2020-01-13T01:46:18Z

model.py

+
+
+def create_encoder(step, data_inputs):
+    encoder_cell = create_stacked_rnn(step)


The create_stacked_rnn was created outside the function in the decoder. When we'll fix the decoder to be used once only, it'll be pushed into it.

guillaume-chevalier

2nd code review pass

guillaume-chevalier · 2020-01-13T03:03:48Z

model.py

+                create_graph=create_graph,
+                create_loss=create_loss,
+                create_optimizer=create_optimizer,
+                create_feed_dict=create_feed_dict


What does this do? I need to understand everything

guillaume-chevalier · 2020-01-13T03:04:40Z

model.py

+    if exercise == 3:
+        generate_x_y_data = generate_x_y_data_v3
+    if exercise == 4:
+        generate_x_y_data = generate_x_y_data_v4


Let's keep the 4 examples. Good.

guillaume-chevalier · 2020-01-13T03:05:20Z

model.py

+
+    pipeline = DeepLearningPipeline(
+        SignalPredictionPipeline(),
+        validation_size=0.15,


There was no validation in the original example. This seems like it is a problem here (and in the LSTM example, too).

guillaume-chevalier · 2020-01-13T03:09:17Z

model.py

+        scoring_function=to_numpy_metric_wrapper(mean_squared_error)
+    )
+
+    data_inputs, expected_outputs = generate_x_y_data(isTrain=True, batch_size=SignalPredictionPipeline.BATCH_SIZE)


I'm not sure what happens here so the following may be wrong but I think I spotted a problem:

I think your problem is that I was calling the generate_x_y_data function at each epoch. You'd probably need here a train and a test data generator class that lazy loads each batches for the DL Minibatching Pipeline.

You'd also need to split the data on the time axis like this here if using cross validation:

So the users could optimize on the validation data with a manual search and manual parameter tuning (no random search), and then call a function to test on the test set or something. So we need to probably change the generate_x_y_data method.

the cross validation is done in the deep learning pipeline from neuraxle.

oh wait you don't have all my changes I actually have a loop that calls generate_x_y_date a few times :/

let's change generate_x_y_data_v4 it was confusing

also the normalization, and lazy data loading should be done in neuraxle

guillaume-chevalier · 2020-01-13T03:10:30Z

model.py

+    mse_train = pipeline.get_epoch_metric_train('mse')
+    mse_validation = pipeline.get_epoch_metric_validation('mse')
+
+    plot_metric(mse_train, mse_validation, xlabel='epoch', ylabel='mse', title='Model Mean Squared Error')


Could you please upload a preview of that plot / chart in your PR for me to inspect this without running the code locally yet?

guillaume-chevalier · 2020-01-13T03:11:55Z

requirements.txt

@@ -0,0 +1,3 @@
+tensorflow==1.14
+-e git://github.com/alexbrillant/Neuraxle.git@a270fe2b2f73c9350d76fcf4b6f058b764a8c8f7#egg=neuraxle
+requests


@alexbrillant Let's change requests to urllib. Let's not import requests anymore. See how I use urllib here:
https://www.neuraxle.org/stable/rest_api_serving.html#API-Call-Example

alexbrillant added 5 commits January 7, 2020 13:03

Neuraxle Refactor

901a1fb

Neuraxle Refactor

4396a76

Remove Old Comptability Code

f6f673b

Dynamic RNN Encoder Decoder Wip With Neuraxle

0b3e9b6

Remove Outdated Notebook

6a0210c

alexbrillant commented Jan 8, 2020

View reviewed changes

model.py Show resolved Hide resolved

alexbrillant changed the title ~~Neuraxle refactor (model.py only other changes were not wanted...)~~ WIP Neuraxle refactor (model.py only other changes were not wanted...) Jan 8, 2020

alexbrillant added 4 commits January 8, 2020 14:16

Remove Unwanted changes to the readme

969bf70

Remove Unwanted Notebook

783ee2f

Remove Unwanted Changes To Old Seq 2 Seq

8c6bc8d

Remove Unwanted Changes To Seq 2 Seq

4f7afd8

alexbrillant changed the title ~~WIP Neuraxle refactor (model.py only other changes were not wanted...)~~ WIP Neuraxle refactor Jan 8, 2020

alexbrillant added 4 commits January 11, 2020 10:27

Use Deep Learning Pipeline, and add metrics

5f1c5a5

Add Output size to maximum_iterations to fix Inference Helper that wa…

c148534

…s infinitely decoding

Fix Validation Size And Scoring Function

49f7478

Extract Class For Signal Prediction Pipeline, And Add Plotting

2e78ecd

guillaume-chevalier requested changes Jan 13, 2020

View reviewed changes

alexbrillant closed this Jan 13, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP Neuraxle refactor #16

WIP Neuraxle refactor #16

alexbrillant commented Jan 8, 2020 •

edited

guillaume-chevalier commented Jan 13, 2020

guillaume-chevalier left a comment

guillaume-chevalier Jan 13, 2020

guillaume-chevalier Jan 13, 2020

guillaume-chevalier Jan 13, 2020

guillaume-chevalier Jan 13, 2020

guillaume-chevalier Jan 13, 2020

guillaume-chevalier Jan 13, 2020

guillaume-chevalier Jan 13, 2020

guillaume-chevalier Jan 13, 2020

guillaume-chevalier Jan 13, 2020

guillaume-chevalier Jan 13, 2020

guillaume-chevalier Jan 13, 2020

guillaume-chevalier Jan 13, 2020

guillaume-chevalier left a comment

guillaume-chevalier Jan 13, 2020

guillaume-chevalier Jan 13, 2020

guillaume-chevalier Jan 13, 2020

guillaume-chevalier Jan 13, 2020

alexbrillant Jan 13, 2020 •

edited

alexbrillant Jan 13, 2020

alexbrillant Jan 13, 2020

alexbrillant Jan 13, 2020

guillaume-chevalier Jan 13, 2020

guillaume-chevalier Jan 13, 2020

	# shape: (batch_size, seq_length, input_dim)
	# shape: (batch_size, seq_length, output_dim)

		# shape: (batch_size)
		target_sequence_length = tf.placeholder(dtype=tf.int32, shape=[None], name='expected_outputs_length')

		return decoder_outputs_training, decoder_outputs_inference


		def create_encoder(step, data_inputs):



		def create_inference_decoder(step: TensorflowV1ModelStep, encoder_state, decoder_cell):
		start_inputs = tf.constant(GO_TOKEN, shape=[step.hyperparams['batch_size'], step.hyperparams['output_dim']])

		return output


		def create_decoder_outputs(step, helper, encoder_state, decoder_cell):



		def create_encoder(step, data_inputs):
		encoder_cell = create_stacked_rnn(step)

WIP Neuraxle refactor #16

WIP Neuraxle refactor #16

Conversation

alexbrillant commented Jan 8, 2020 • edited

guillaume-chevalier commented Jan 13, 2020

guillaume-chevalier left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guillaume-chevalier left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexbrillant Jan 13, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alexbrillant commented Jan 8, 2020 •

edited

alexbrillant Jan 13, 2020 •

edited