Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Keras materializer to work with remote artifact stores #569

Merged
merged 13 commits into from
May 10, 2022

Conversation

safoinme
Copy link
Contributor

@safoinme safoinme commented May 6, 2022

Describe changes

Change the Keras model materializer to support the Azure artifact store. By saving the model in a temp file then use fileio to copy it to remote storage.

        with tempfile.NamedTemporaryFile(
            mode="w", suffix=".hdf5", delete=True
        ) as f:
            model.save(f.name)
            fileio.copy(f.name, filepath)

Pre-requisites

Please ensure you have done the following:

  • I have read the CONTRIBUTING.md document.
  • If my change requires a change to docs, I have updated the documentation accordingly.
  • If I have added an integration, I have updated the integrations table.
  • I have added tests to cover my changes.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Other (add details above)

@github-actions github-actions bot added internal To filter out internal PRs and issues bug Something isn't working labels May 6, 2022
@safoinme safoinme changed the base branch from main to develop May 6, 2022 20:33
@safoinme safoinme changed the title Bugfix/keras materializer with azure Fix Keras Materializer to work with azure artifact store May 6, 2022
@safoinme safoinme requested a review from htahir1 May 6, 2022 20:35
@safoinme safoinme changed the title Fix Keras Materializer to work with azure artifact store Fix Keras materializer to work with Azure artifact store May 6, 2022
Copy link
Contributor

@htahir1 htahir1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a cool change 🚀. I'm unsure whether the PR title is accurate though. It not only works with Azure store but also all remote stores now right?

@@ -119,7 +122,7 @@ def visualize_statistics():
# Run the pipeline
p = boston_housing_pipeline(
importer=importer(),
trainer=trainer(),
trainer=trainer().with_return_materializers(KerasMaterializer),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need this here?

Copy link
Contributor Author

@safoinme safoinme May 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's for every remote storage.
Sorry I will remove it, we don't need it actually

@safoinme safoinme changed the title Fix Keras materializer to work with Azure artifact store Fix Keras materializer to work with remote artifact stores May 6, 2022
Copy link
Contributor

@htahir1 htahir1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@safoinme safoinme requested a review from bcdurak May 8, 2022 18:45
Copy link
Contributor

@bcdurak bcdurak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing changes. Everything looks great

@safoinme safoinme merged commit 14c2b42 into develop May 10, 2022
@safoinme safoinme deleted the bugfix/keras-materializer-with-azure branch May 10, 2022 10:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working internal To filter out internal PRs and issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants