Techniques to run MONAI Deploy App in batch mode or in a long running service

[MMelQin]

Questions have been asked

• how can I run the app to process a batch of payloads, keeping the loaded model network in memory in order to have better perf?
• how can I run the app as a service?
• why does the app exist once it has processed just one input payload?

Well, we deliberately leave the run mode decision to the users/app developers, taking advantage of the already supported features of the App SDK Application.

The Application executor internally maintains a execution index, and each call to run() will be assigned a new index, making it re-runnable so long as the code calling the Application.run() keeps the app object alive.

In the following simplest example, I am showing a quick change to the __main__ function of the Spleen app to process payloads contained in subfolders in the input path, with each run of the app processing one and only one subfolder's content as the discrete payload (though may be multiple DICOM instances). The same app object with PyTorch model network is kept in memory and reused. The output for each run is saved in its own folder under the app output_path.

One can also create a service and run the Application to process continuous input: make the __main__ function a main loop, with a file monitor, queue listener, interrupt handler, or other means, on-trigger move the input to the app's input folder, call the app.run() and then handle/move the output.

Note that "multi-threading" to run app has not been tested and is not advised for the time being. This will be tested to ensure no contention or undesirable side effect regarding the input_path in the app object. So, please serialize the call to run().

if __name__ == "__main__":
    # Creates the app and test it standalone. When running is this mode, please note the following:
    #     -m <model file>, for model file path
    #     -i <DICOM folder>, for input DICOM CT series folder
    #     -o <output folder>, for the output folder, default $PWD/output
    # e.g.
    #     monai-deploy exec app.py -i input -m model/model.ts
    #

    from pathlib import Path

    # This main function is an example to show how a batch of input can be processed.
    # It assumes that in the app input folder there are a number of subfolders, each
    # containing a discrete input to be processed. Each discrete payload can have
    # multiple DICOM instances file, optionally organized in its own folder structure.
    # The application object is first created, and on its init the model network is
    # loaded as well as pre and post processing transforms. This app object is then
    # run multiple times, each time with a single discrete payload.

    app = AISpleenSegApp(do_run=False)

    # Preserve the application top level input and output folder path, as the path
    # in the context may change on each run if the I/O arguments are passed in.
    app_input_path = Path(app.context.input_path)
    app_output_path = Path(app.context.output_path)

    # Get subfolders in the input path, assume each one contains a discrete payload
    input_dirs = [path for path in app_input_path.iterdir() if path.is_dir()]

    # Set the output path for each run under the app's output path, and do run
    for d in input_dirs:
        try:
            output_path = app_output_path / f"{d.name}_output"
            logging.info(
                f"Start processing input in: {d} with results in: {output_path}"
            )

            # Run app with specific input and output path.
            # Passing in the input and output do have the side effect of changing
            # app context. This side effect will likely be eliminated in later releases.
            app.run(
                input=d,
                output=output_path,
            )

            logging.info(f"Completed processing input in: {d} with results in: {output_path}")
        except Exception as ex:
            logging.error(f"Failed processing input in {d}, due to: {ex}\n")

[MMelQin]

@aihsani @nakfour @CPBridge This will be useful for some of your use cases.

[ericspod]

The idea we discussed today was to change the entry point for MAP images to be a new Python based program which can run in a command line mode but also as a server using Ray or FastAPI to serve access to the model in a lightweight manner. This would be optional and something that wouldn't interfere with the current mode of using MAPs in other environments eg. Argo. Generating code to make use of Ray/FastAPI/BentoML would be relatively straight forward in addition to the existing main program.

[MMelQin]

Thanks @ericspod .I've also created a "Show and tell" topic #340 with some details on the metadata of the MAP (container image) and running MAP as a Docker container with a simple shell script, without any need of App SDK CLI or MONAI Deploy platform. It can be used as a reference for certain use cases.