[BUG]: StepInterfaceError: Unable to find materializer for output 'output' of type <class 'keras.engine.training.Model'> in step 'trainer'.
#874
Comments
|
I am getting the same error for my simple code. from zenml.steps import step, Output
@step
def data_loader() -> Output(x_train=np.ndarray, y_train=np.ndarray, x_test=np.ndarray, y_test=np.ndarray):
"""
return MNIST x_train, y_train, x_test, y_test
"""
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
return x_train, y_train, x_test, y_test
@step
def initialize_model() -> Output(untrained_model=tf.keras.Sequential):
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
tf.keras.layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
tf.keras.layers.Flatten(),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(num_classes, activation="softmax"),
])
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['acc'])
return model
@step
def preprocess_data(x:np.ndarray, y:np.ndarray) ->Output(x_processed= np.ndarray, y_processed=np.ndarray):
x_processed = x.astype("float32") / 255
x_processed = np.expand_dims(x_processed, -1)
y_processed = keras.utils.to_categorical(y, num_classes)
return x_processed, y_processed
@step
def train_model(x_train:np.ndarray, y_train:np.ndarray, untrained_model:tf.keras.Sequential) -> Output(trained_model=tf.keras.Sequential, history=dict):
history = untrained_model.fit(x_train, y_train, epochs=epochs)
return untrained_model, history
@step
def eval_model(trained_model: tf.keras.Sequential, x_test:np.ndarray, y_test:np.ndarray) -> None:
trainedmodel.evaluate(x_test, y_test)
from zenml.pipelines import pipeline
@pipeline
def runner(data_loader, initialize_model, preprocess_data, train_model):
x_train, y_train, x_test, y_test = data_loader()
x_train_processed, y_train_processed = preprocess_data(x_train, y_train)
model = initialize_model()
model, history = train_model(x_train_processed, y_train_processed, model)
mnist_pipeline = runner(data_loader(), initialize_model(), preprocess_data(), train_model())
mnist_pipeline.run()The error is |
|
@ahmadmustafaanis the error you are getting is because there's no materializer written for type The code should run if you replace |
|
@dudeperf3ct I am still getting this error. the function |
|
@ahmadmustafaanis what version of zenml are you using? I am able to run the pipeline using the following code using zenml 0.13.2. import numpy as np
import tensorflow as tf
from zenml.steps import step, Output
num_classes = 10
epochs = 10
@step
def data_loader() -> Output(x_train=np.ndarray, y_train=np.ndarray, x_test=np.ndarray, y_test=np.ndarray):
"""
return MNIST x_train, y_train, x_test, y_test
"""
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
return x_train, y_train, x_test, y_test
@step
def initialize_model() -> Output(untrained_model=tf.keras.Model):
model = tf.keras.Sequential([
tf.keras.layers.Conv2D(32, kernel_size=(3, 3), activation="relu", input_shape=(28, 28, 1)),
tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
tf.keras.layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
tf.keras.layers.MaxPooling2D(pool_size=(2, 2)),
tf.keras.layers.Flatten(),
tf.keras.layers.Dropout(0.5),
tf.keras.layers.Dense(num_classes, activation="softmax"),
])
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['acc'])
return model
@step
def preprocess_data(x:np.ndarray, y:np.ndarray) -> Output(x_processed= np.ndarray, y_processed=np.ndarray):
x_processed = x.astype("float32") / 255
x_processed = np.expand_dims(x_processed, -1)
y_processed = tf.keras.utils.to_categorical(y, num_classes)
return x_processed, y_processed
@step
def train_model(x_train:np.ndarray, y_train:np.ndarray, untrained_model:tf.keras.Model) -> Output(trained_model=tf.keras.Model, history=dict):
history = untrained_model.fit(x_train, y_train, epochs=epochs)
return untrained_model, history.history
@step
def eval_model(trained_model: tf.keras.Model, x_test:np.ndarray, y_test:np.ndarray) -> None:
trained_model.evaluate(x_test, y_test)
from zenml.pipelines import pipeline
@pipeline
def runner(data_loader, initialize_model, preprocess_data, train_model):
x_train, y_train, x_test, y_test = data_loader()
x_train_processed, y_train_processed = preprocess_data(x_train, y_train)
model = initialize_model()
model, history = train_model(x_train_processed, y_train_processed, model)
mnist_pipeline = runner(data_loader(), initialize_model(), preprocess_data(), train_model())
mnist_pipeline.run() |
|
My version is |
|
@ahmadmustafaanis I'm able to run @dudeperf3ct code with Did you install tensorflow using pip? Or using zenml integration eg |
|
Yes I installed it via pip, not integrated via zenml. I guess that's the issue, I'll try with zenml integration this weekend. |
|
@ahmadmustafaanis quite possibly. Using the |
|
@dnth Thanks, working like charm now. This means that almost every object, that is a output from a zenml step(or input to), needs to be installed via |
|
@ahmadmustafaanis yes, for every object that passes through the pipeline, you need to tell zenml how to serialize/deserialize it. In this way this object can be cached and stored. In zenml this is done using something called a Materializer. For most Python objects you don't need to write your own Materializer, these are already builtin within zenml. But for custom objects you'd need to write a custom materializer for it. Here's a guide how to write a custom materializer A Keras model is not a standard Python object so you'd need to write one. But this has been done by someone in zenml and to use it you'll have to install the integration with |
|
@dudeperf3ct any chance this also solves your issue? :) |
|
@dnth i didn't face any issue ... i was helping in resolving the issue |
|
@dudeperf3ct I see.. sorry I misunderstood! Thank you so much for your help! |
|
@Coder-Vishali please let us know if this solves the issue for you :) |
|
@Coder-Vishali I'm closing this issue due to inactivity. Please let us know if the solution provided worked for you. Feel free to reopen if the problem still persists :) |
Contact Details [Optional]
vishalisrinivasan97@gmail.com
System Information
ZenML version: 0.13.1
Install path: /home//.pyenv/versions/3.7.6/lib/python3.7/site-packages/zenml
Python version: 3.7.6
Platform information: {'os': 'linux', 'linux_distro': 'ubuntu', 'linux_distro_like': 'debian', 'linux_distro_version': '18.04'}
Environment: native
Integrations: ['kubeflow', 'kubernetes', 'tensorboard', 'tensorflow']
What happened?
When I try to implement this code "https://github.com/zenml-io/zenml/tree/main/examples/kubeflow_pipelines_orchestration"; at the step "Run pipeline":
_
_
When I try to run the above snippet, I face StepInterfaceError
StepInterfaceError: Unable to find materializer for output 'output' of type
<class 'keras.engine.training.Model'>in step 'trainer'. Please make sure to either explicitly set a materializer for step outputs using
step.with_return_materializers(...)or registering a default materializer for specific types by subclassingBaseMaterializerand setting itsASSOCIATED_TYPESclass variable. For more information, visithttps://docs.zenml.io/developer-guide/advanced-usage/materializer.
Reproduction steps
You can get the entire source code from here: https://github.com/zenml-io/zenml/tree/main/examples/kubeflow_pipelines_orchestration
Initialize the pipeline
first_pipeline = mnist_pipeline(
importer=importer(),
normalizer=normalizer(),
trainer=trainer(),
evaluator=evaluator(),
)
first_pipeline.run()
Relevant log output
Code of Conduct
The text was updated successfully, but these errors were encountered: