Add MLModel Service for Android

[edaniels]

This introduces an MLModel service abstracted around https://djl.ai/ with an example implementation at android/examples/mlmodle-module. Since this type of code adds considerable size to the final APK, the following modules were added outside the core: viam-core-sdk-mlmodel-service, viam-java-sdk-mlmodel-service, and viam-android-sdk-mlmodel-service. A user wanting to use this in android would depend on viam-android-sdk-mlmodel-service as it has the needed android specific shared objects and runtime dependencies.

Inside a module, you use com.viam.sdk.core.service.mlmodel.Registry.useService(); to register the mlmodel service before constructing a module. This is needed due to the mlmodel java module being outside of the core SDK proper.

NDArrays are backed by an NDManager which can be set as a singleton in com.viam.sdk.core.service.mlmodel.MLModelRPCService#setNDManager. If not set, NDManager.newBaseManager() is used and it is passed into infer methods.

More on the example:

• MNIST model trained by DLJ
• Input of a HWC (the actual model expects a tensor of [1,28,28])
• Output 10 softmaxed probabilities from 0-9 (the actual model outputs a [1,10])

Some python code to run against this if you have an image (there's one in the example resources directory):

import numpy as np

import asyncio

from viam.robot.client import RobotClient
from viam.services.mlmodel.client import MLModelClient
from viam.rpc.dial import Credentials, DialOptions
import cv2


async def connect():
    opts = RobotClient.Options.with_api_key(
      api_key='key',
      api_key_id='id'
    )
    return await RobotClient.at_address('address', opts)

async def main():
    img = cv2.imread('/0.png').astype(np.float32)
    robot = await connect()

    my_mlmodel = MLModelClient.from_robot(robot=robot, name="mlmodel1")

    input_tensors = {"image": img}

    metadata = await my_mlmodel.metadata()
    print(metadata)

    output_tensors = await my_mlmodel.infer(input_tensors)
    print(output_tensors['classifications'])


    # Don't forget to close the machine when you're done!
    await robot.close()

if __name__ == '__main__':
    asyncio.run(main())

[bhaney]

Inital questions and comments - there aren't that many! I don't know java very well but the ML stuff is looking good. I'm going to test out your example first before I approve

[bhaney]

This is a nice cute function - takes care of all the re-shaping and RGB -> grayscale stuff

[edaniels]

yeah it's basically magic to me

[bhaney]

Just for my own knowledge, could you tell me more about the manager, and its function? What is it managing, and why does it have to be an argument to the module?

[edaniels]

A great explanation is here https://docs.djl.ai/docs/development/memory_management.html. I don't love it but basically we need it with this library. It's managing all the memory for NDArrays. I assume it handles pooling and what not. It also handles CPU <-> GPU memory transfers (and other Devices)

[bhaney]

Are all of these put commands copies of the data? No way to just assign the pointer to the dataList into the buffer?

[edaniels]

On the from proto to ndarray side, no, unfortunately unless there's some special version of NDManager that can do it which I haven't found.

[edaniels]

FWIW, the other libraries I used as well were also opinionated about copying data in.

[bhaney]

I understand the copy to proto has to happen, but I'm curious is there is a way to keep from having to copy the data from the Proto message?

[edaniels]

I think these two comments are mixed up on directions since this one is to proto. For the toProto case, there's no clear way to set the data to a pointer other than to implement your own serializer. I think if we're worried about performance, it should be measured first. The NDManager may be good at being speedy and a good steward of memory usage.

[bhaney]

Yea, didn't mean to put the comment on this one. I'm more worried about proto -> ndArray, as it seems like you could just use the data in the proto struct directly and copying would be unecessary. The NDManager does sound like a good lead if it turns out to be a problem

[bhaney]

Thanks for this! LGTM!