Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Export Tensorflow SavedModel? #2991

Closed
AidanNelson opened this issue Nov 27, 2019 · 11 comments
Closed

Export Tensorflow SavedModel? #2991

AidanNelson opened this issue Nov 27, 2019 · 11 comments
Assignees
Labels
discussion stale

Comments

@AidanNelson
Copy link

@AidanNelson AidanNelson commented Nov 27, 2019

Hi There,

I am exploring the possibility of training a model in Unity using the ML-Agents package, then exporting that model for use with Tensorflow.js in the browser. Eventually, I'd like to incorporate that model into ml5.js, an approachable wrapper library around tf.js. I think this workflow would be an exciting use of the training / curriculum learning structure of Unity, and allow people to create models which could be used in online demonstrations of reinforcement learning (...or maybe agent-based AI for online games?)

Currently, it is possible to convert Tensorflow models for use with Tensorflow.js as long as they are in the SavedModel, Keras model, or TensorFlow Hub module format.

Describe the solution you'd like
I would like to be able to export a trained model from Unity in one of the above Tensorflow formats (SavedModel, Keras model, or TensorFlow Hub module), for later conversion to Tensorflow.js using existing solutions.

I see that in the current release of ML-Agents, Barracuda is used to store models, and that a Tensorflow to Barracuda conversion script exists, but am unsure how possible it would be to convert a model from Barracuda back to Tensorflow and the extent of this work.

Any information about this would be most appreciated!

Aidan

A (vaguely) related issue:
#1802

UPDATE (2019.12.2)
I looked further and realized that Unity Ml-Agents still uses Tensorflow for training, and only converts the models to Barracuda '.nn' files on export for inference. Now I am trying to figure out how to export a SavedModel format in this export_model function, for later conversion to TFJS-capable model. Does anyone have experience with using the saved model builder and a sense for how it could be implemented here? I've altered the export_model function to use tf.compat.v1.saved_model.Builder in the following code:

def export_model(self):
        """
        Exports latest saved model to .nn format for Unity embedding.
        """
        with self.graph.as_default():
  
            # MY ADDITIONAL CODE:
            builder = tf.compat.v1.saved_model.Builder(self.model_path + "/SavedModel/")
            builder.add_meta_graph_and_variables(self.sess,
                                                [tf.saved_model.tag_constants.TRAINING],
                                                strip_default_attrs=True)
            builder.add_meta_graph([tf.saved_model.tag_constants.SERVING], strip_default_attrs=True)
            builder.save()
            # END OF ADDITIONAL CODE
                                
           target_nodes = ",".join(self._process_graph())
           graph_def = self.graph.as_graph_def()
            
            output_graph_def = graph_util.convert_variables_to_constants(
                self.sess, graph_def, target_nodes.replace(" ", "").split(",")
            )
            frozen_graph_def_path = self.model_path + "/frozen_graph_def.pb"
            with gfile.GFile(frozen_graph_def_path, "wb") as f:
                f.write(output_graph_def.SerializeToString())
            tf2bc.convert(frozen_graph_def_path, self.model_path + ".nn")
            logger.info("Exported " + self.model_path + ".nn file")

This runs without error, but produces a SavedModel without any Signature Definitions, as seen when I run the python saved_model_cli.py show --dir ./ --all as per this:

MetaGraphDef with tag-set: 'serve' contains the following SignatureDefs:

MetaGraphDef with tag-set: 'train' contains the following SignatureDefs:

This then throws an error when converting to TFJS using the tfjs-converter:

ValueError: Signature 'serving_default' does not exist. The following signatures are available: KeysView(_SignatureMap({}))

along with many warnings like this:

WARNING:tensorflow:Unable to create a python object for variable <tf.Variable 'beta2_power:0' shape=() dtype=float32_ref> because it is a reference variable. It may not be visible to training APIs. If this is a problem, consider rebuilding the SavedModel after running tf.compat.v1.enable_resource_variables().

If anyone can shed light on how to include SignatureDefs in the exported model, and export a valid SavedModel from Unity ML-Agents, that would be awesome!

Thanks again.

@AidanNelson AidanNelson added the request label Nov 27, 2019
@AidanNelson AidanNelson changed the title Export Barracuda Model to Tensorflow Model Export Tensorflow SavedModel? Dec 2, 2019
@unityjeffrey unityjeffrey self-assigned this Dec 2, 2019
@unityjeffrey
Copy link
Contributor

@unityjeffrey unityjeffrey commented Dec 2, 2019

Thank you for your comments. We’ve documented your feedback and will prioritize when appropriate.

@unityjeffrey unityjeffrey added discussion and removed request labels Dec 2, 2019
@unityjeffrey
Copy link
Contributor

@unityjeffrey unityjeffrey commented Dec 2, 2019

also - opening this up as a more general discussion to see if folks in the community have attempted to do what you are describing above.

@AidanNelson
Copy link
Author

@AidanNelson AidanNelson commented Dec 9, 2019

For anyone trying to use models from Unity in TensorFlow.js, here is the approach I took.

  • Install Unity ML-Agents for development by following this guide here
  • In ml-agents/mlagents/trainers/tf_policy.py script, add the following lines to the export_model function to export a TensorFlow SavedModel. Note that the required graph nodes are different for continuous action space (i.e. the agent takes action on float values) and discrete action space (i.e. the agent takes action on integer values), so you will need to uncomment the respective lines accordingly:
def export_model(self):
        """
        Exports latest saved model to .nn format for Unity embedding.
        """

        with self.graph.as_default():
            graph_def = self.graph.as_graph_def()

            # BEGINNING OF ADDED CODE:
            # To learn more about nodes of the graph, uncomment the following lines:
            # for node in graph_def.node:
            #     print("-------")
            #     print(node.name)
            #     print(node.input)
            #     print(node.attr)
 
            # Uncomment for discrete vector action space:
            # vectorInputNode = self.graph.get_tensor_by_name("vector_observation:0")
            # actionMaskInput = self.graph.get_tensor_by_name("action_masks:0")
            # actionOutputNode = self.graph.get_tensor_by_name("action:0")
            # sigs = {}
            # sigs[tf.compat.v1.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY] = \
                tf.saved_model.signature_def_utils.predict_signature_def( \
                    {"in": vectorInputNode, "actionMask": actionMaskInput}, {"out": actionOutputNode})
            
            # Uncomment for continuous vector action space:
            # vectorInputNode = self.graph.get_tensor_by_name("vector_observation:0")
            # epsilonInputNode = self.graph.get_tensor_by_name("epsilon:0")
            # actionOutputNode = self.graph.get_tensor_by_name("action:0")
            # sigs = {}
            # sigs[tf.compat.v1.saved_model.signature_constants.DEFAULT_SERVING_SIGNATURE_DEF_KEY] = \
            #     tf.saved_model.signature_def_utils.predict_signature_def( \
            #         {"in": vectorInputNode,"epsilon": epsilonInputNode}, {"out": actionOutputNode})



            
            builder = tf.compat.v1.saved_model.Builder(self.model_path + "/SavedModel/")
            builder.add_meta_graph_and_variables( \
                self.sess, \
                [tf.saved_model.tag_constants.SERVING], \
                signature_def_map=sigs, \
                strip_default_attrs=True)
            builder.save()                                
            # END OF ADDED CODE
           
            target_nodes = ",".join(self._process_graph())
            
            output_graph_def = graph_util.convert_variables_to_constants(
                self.sess, graph_def, target_nodes.replace(" ", "").split(",")
            )
            frozen_graph_def_path = self.model_path + "/frozen_graph_def.pb"
            with gfile.GFile(frozen_graph_def_path, "wb") as f:
                f.write(output_graph_def.SerializeToString())
            tf2bc.convert(frozen_graph_def_path, self.model_path + ".nn")
            logger.info("Exported " + self.model_path + ".nn file")
     
  • the exported SavedModel should be able to be converted to a web model by the latest tensorflowjs-converter (tfjs version 1.4.0, currently)

If anyone can clarify the different output nodes: action vs. action_probs/action_probs, please let me know!

@stale
Copy link

@stale stale bot commented Dec 23, 2019

This issue has been automatically marked as stale because it has not had activity in the last 14 days. It will be closed in the next 14 days if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Dec 23, 2019
@stale
Copy link

@stale stale bot commented Jan 6, 2020

This issue has been automatically closed because it has not had activity in the last 28 days. If this issue is still valid, please ping a maintainer. Thank you for your contributions.

@stale stale bot closed this Jan 6, 2020
@AidanNelson
Copy link
Author

@AidanNelson AidanNelson commented Jun 18, 2020

@unityjeffrey I would love to open this thread up again now that MLAgents is in an official release.

The ability to export models as Tensorflow SavedModel format would allow for interoperability with Tensorflow.js, a very well supported web-based inference engine with a large community around it. The ability to use models trained in Unity on the web (even outside of the Unity ecosystem on the web) would open up a lot of possibilities for Unity ML Agents as a training platform. The advantages of web-based inference are numerous. My particular interest is in creating web based educational tools around machine learning.

Here is a model trained in Unity, then exported as a Tensorflow SavedModel, converted for use with Tensorflow.js, and loaded in a javascript sketch: https://www.aidanjnelson.com/balance-ball/

Thoughts?

@unityjeffrey
Copy link
Contributor

@unityjeffrey unityjeffrey commented Jun 18, 2020

hi @AidanNelson - quick question for you. Can you describe a use case where you would want to run an ML-Agents model via Tensorflow.js? Barracuda allows for ML-Agents models to be run in WebGL games.

@AidanNelson
Copy link
Author

@AidanNelson AidanNelson commented Jun 18, 2020

Sure!

Part of what is great about the Unity ML-Agents Framework is the ability to quickly spin up a training environment and get started experimenting with RL without a deep knowledge of the difference between SAC and PPO or the ability to tune the various training hyperparameters. ML-Agents provides an easier (and more powerful) access point to training RL than I've seen elsewhere, in an environment which many people are already familiar with. That said, there are a number of contexts in which someone might want the final version of their project to exist online outside of a Unity WebGL build. The first one that comes to mind is a classroom environment.

Teachers teaching Javascript (in a middle or high school environment, for instance) with the ability to incorporate a simple RL model into their curriculum would be able to:

  • live-code on an environment in javascript and see results without needing to first compile,
  • have students build applications using these models without needing to know how to use Unity,
  • have students build applications using these models without needing to have Unity installed on their machines (in computer lab environment, for example),
  • interact with other Javascript libraries / frameworks / browser APIs (webcam, etc.) in a familiar manner,
  • host javascript sketches on various platforms (p5.js web editor, Glitch, etc.)

Personally, I want to work with incorporating models into projects using p5.js and ml5.js, libraries which have large and welcoming communities around them and are specifically geared towards accessibility and education.

Any thoughts?

@unityjeffrey
Copy link
Contributor

@unityjeffrey unityjeffrey commented Jun 18, 2020

Hi @AidanNelson - it feels like ML-A + Unity would not be a good solution for what you are describing (teaching Javascript with a small RL component). I think there may be more straightforward examples like uploading a picture and doing some image classification. However, let me know if I am missing some additional context.

@AidanNelson
Copy link
Author

@AidanNelson AidanNelson commented Jun 22, 2020

I think there may be more straightforward examples like uploading a picture and doing some image classification. However, let me know if I am missing some additional context.

@unityjeffrey - You're definitely right that there are examples of browser-based ML which are more straightforward to implement. Image Classification, doodle classifiers, etc. are all available as part of ml5.js or TFJS directly and very easy to implement. From a student perspective, however, reinforcement learning presents a really valuable opportunity for learning ML because an 'agent' whose actions are being controlled by an RL brain is a much more visible analogy than traditional ML 'Hello World' projects. What I mean is that the 'decision making' process happens in real time and happens over time. Rather than being presented with a single input and an output, we are presented with a continuous series of inputs and outputs, which, taken in aggregate, seems to form the 'behavior' of an agent. Because of this, RL is also more fun!

Reinforcement Learning has (up until now) been quite challenging to implement in the browser.
For context, the browser-based RL libraries which include training capabilities (ConvnetJS, REINFORCEjs, among others) seem mostly to be stale. Tensorflow.js does include some 'official' RL examples (CartPole and SnakeDQN), which are exciting examples but difficult to train / implement. This is an area in which Unity ML Agents would allow teachers / curriculum developers to develop custom models for specific demonstrations.

ML-Agents could really lower the barrier to entry for teaching about RL, and I think interoperability with TFJS would help that.

I think this is just one of numerous places where having a Unity-trained RL model in the browser (outside of a WebGL build) would be exciting and fun. I am happy to brainstorm others (the first one that comes to mind is having an agent interact directly with DOM elements) if it would be helpful for this feature-request / discussion.

@github-actions
Copy link

@github-actions github-actions bot commented Jan 30, 2021

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 30, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
discussion stale
Projects
None yet
Development

No branches or pull requests

2 participants