Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

# 1 Multi source aggregation & Refactor Mixins #246

Merged

Conversation

@alexbrillant
Copy link
Member

alexbrillant commented Jan 28, 2020

In ValidationSplitWrapper:

train_data_container, validation_data_container = self.split_data_container(
    data_container
)

# add sub data container for the validation metrics 
# that will be calculated in the MetricsWrapper step ! 
train_data_container.add_sub_data_container(
    name='validation', 
    sub_data_container=validation_data_container
)

In MetricsWrapper:

if 'validation' in data_container:
     self.set_train(False)

     validation_data_container = self._transform_data_container(
           data_container['validation'], 
           context
     )
     self._did_fit_transform_or_transform(validation_data_container, context)

     self.set_train(True)

New ZipData step ! :

# 3d shape: (batch_size, time_steps, n_features)
# 2d shape: (batch_size, batch_features)
# 1d shape: (batch_size)
data_container = DataContainer(
    data_inputs=data_inputs_3d, 
    expected_outputs=expected_outputs_3d
) .add_sub_data_container('1d_data_source', data_container_1d)   \ 
  .add_sub_data_container('2d_data_source', data_container_2d)

# When
p = Pipeline([
   ZipData(data_sources=['1d_data_source', '2d_data_source'])
])

data_container = p.handle_transform(data_container, ExecutionContext())

# new_shape: (batch_size, time_steps, n_features + batch_features + 1)
@cla-bot cla-bot bot added the cla-signed label Jan 28, 2020
@alexbrillant alexbrillant changed the title (WIP) Multi source aggregation : sub data containers ! composite design pattern (WIP merge deep learning pipeline pull request before this one thanks) Multi source aggregation : sub data containers ! composite design pattern Jan 28, 2020
… to 3d, and 1d to 2d
@alexbrillant alexbrillant changed the title (WIP merge deep learning pipeline pull request before this one thanks) Multi source aggregation : sub data containers ! composite design pattern # 1 Multi source aggregation : sub data containers ! composite design pattern Jan 30, 2020
@alexbrillant

This comment has been minimized.

Copy link
Member Author

alexbrillant commented Jan 30, 2020

@guillaume-chevalier needs to be merged after the deep learning pipeline review... contains important changes for passing validation for calculating the metrics anywhere. also contains more changes. sry I didn't split this perfectly. however, it shouldn't be too bad if we merge the deep learning pipeline review before this one.

@alexbrillant alexbrillant changed the title # 1 Multi source aggregation : sub data containers ! composite design pattern # 2 Multi source aggregation : sub data containers ! composite design pattern Jan 30, 2020
@alexbrillant alexbrillant changed the title # 1 Multi source aggregation : sub data containers ! composite design pattern # 1 Multi source aggregation Feb 8, 2020
Copy link
Member

guillaume-chevalier left a comment

Looks finally good! After the review of alexbrillant#1

Just be sure to fix the last Shouldn't sub_data_containers be a dict rather than a tuple list? comment above that isn't resolved yet before merging.

self.set_expected_outputs(new_expected_outputs)
self.set_current_ids(list(self.current_ids))
self.set_data_inputs(list(self.data_inputs))
self.set_expected_outputs(list(self.expected_outputs))

This comment has been minimized.

Copy link
@guillaume-chevalier

guillaume-chevalier Feb 8, 2020

Member

This will bug if self.expected_outputs is None. We should probably do a if on each of those 3 values above so that if they are none, they remain None. Is this a real thing? Does what I say is useful?

@alexbrillant alexbrillant changed the title # 1 Multi source aggregation # 1 Multi source aggregation & Refactor Mixins Feb 12, 2020
@guillaume-chevalier guillaume-chevalier merged commit cb54862 into Neuraxio:master Feb 16, 2020
3 checks passed
3 checks passed
build (3.6)
Details
build (3.7)
Details
verification/cla-signed
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.