Platform for Situated Intelligence is an open, extensible framework that enables the development, fielding and study of multimodal, integrative-AI systems.
In recent years, we have seen significant progress with machine learning techniques on various perceptual and control problems. At the same time, building end-to-end, multimodal, integrative-AI systems that leverage multiple technologies and act autonomously or interact with people in the open world remains a challenging, error-prone and time-consuming engineering task. Numerous challenges stem from the sheer complexity of these systems and are amplified by the lack of appropriate infrastructure and development tools.
The Platform for Situated Intelligence project aims to address these issues and provide a basis for developing, fielding and studying multimodal, integrative-AI systems. The platform consists of three layers. The Runtime layer provides a parallel programming model centered around temporal streams of data, and enables easy development of components and applications using .NET, while retaining the performance properties of natively written, carefully tuned systems. A set of Tools enable multimodal data visualization, annotations, analytics, tuning and machine learning scenarios. Finally, an open ecosystem of Components encapsulate various AI technologies and allow for quick compositing of integrative-AI applications.
For more information about the goals of the project, the types of systems that you can build using it, and the various layers see Platform for Situated Intelligence Overview.
Platform for Situated Intelligence is built on the .NET Framework. Large parts of it are built on .NET Standard and therefore run both on Windows and Linux, whereas some components are specific and available only to one operating system.
You can build applications based on Platform for Situated Intelligence either by leveraging nuget packages, or by cloning and building the code. Below are instructions:
The documentation for Platform for Situated Intelligence is available in the github project wiki. The documentation is still under construction and in various phases of completion. If you need further explanation in any area, please open an issue and label it documentation, as this will help us target our documentation development efforts to the highest priority needs.
Getting Started. We recommend starting with the Brief Introduction tutorial, which provides a guided walk-through for some of the main concepts in \psi. It shows how to create a simple \psi application, describes the core concept of a stream, and explains how to transform, synchronize, visualize, persist to and replay streams from disk. We recommend that you first work through the examples in the Brief Introduction to familiarize yourself with these core concepts, before you peruse the other available tutorials. Two other helpful tutorials if you are just getting started are the Writing Components tutorial, which explains how to write new \psi components, and the Delivery Policies tutorial, which describes how to control throughput on streams in your application.
Advanced Topics. A set of documents on more advanced topics describe in more detail various aspects of the framework, including stream fusion and merging, interpolation and sampling, windowing operators, remoting, interop, shared objects and memory management, etc.
Samples. Besides the tutorials and topics, we also recommend looking through the set of Samples provided. While some of the samples address specialized topics such as how to leverage speech recognition components or how to bridge to ROS, reading them will give you more insight into programming with \psi.
Components. Additional useful information regarding available packages and components can be found in the NuGet packages list and in the component list pages. The latter page also has pointers to other repositories by third parties containing other \psi components.
API Reference. An additional API Reference is also available.
If you find a reproducible bug or if you would like to request a new feature or additional documentation, please file an issue on the github repo. If you do so, please first check whether a corresponding issue has already been filed. Use the bug label when filing issues that represent code defects, and provide enough information to reproduce. Use the feature request label to request new features, and use the documentation label to request additional documentation.
We hope the community can help improve and evolve Platform for Situated Intelligence, and we welcome contributions in a variety of forms: from simply using it and filing issues and bugs, to writing and releasing your own new components, to creating pull requests for bug fixes or new features, etc. The Contributing Guidelines page in the wiki describes in more detail a variety of ways in which you can get involved, how the source code is organized, and other useful things to know before starting to make source code changes.
Platform for Situated Intelligence is currently being used in a number of industry and academic research labs, including (but not limited to):
- in the Situated Interaction project, as well as other research projects at Microsoft Research.
- in the MultiComp Lab at Carnegie Mellon University.
- in the Speech Language and Interactive Machines research group at Boise State University.
- in the Qualitative Reasoning Group, Northwestern University.
- in the Intelligent Human Perception Lab, at USC Institute for Creative Technologies.
If you would like to be added to this list, just add a GitHub issue and label it with the whoisusing label. Add a url for your research lab, website or project that you would like us to link to.
The codebase is currently in beta and various aspects of the platform are at different levels of completion and robustness. There are probably still bugs in the code and we will likely be making breaking API changes. We plan to continuously improve the framework and we encourage the community to contribute.
The Roadmap document provides more information about our future plans.
Platform for Situated Intelligence is available under an MIT License. See also Third Party Notices.
We would like to thank our internal collaborators and external early adopters, including (but not limited to): Daniel McDuff, Kael Rowan, Lev Nachmanson and Mike Barnett at MSR, Chirag Raman and Louis-Phillipe Morency in the MultiComp Lab at CMU, as well as adopters in the SLIM research group at Boise State and in the Qualitative Reasoning Group at Northwestern University.