Welcome to the initial stages of the AutoMetic Assistant Project. This ambitious venture aims to build a personalized autonomous multimodal assistant, pushing the boundaries of AI interaction and convenience. Inspired by the advanced PandaGPT framework, our project promises a revolution in digital life management.
The goal for AutoMetic Assistant is to pilot your PC, view your desktop, and complete complex tasks. Through multimodal inputs, it aims to offer an unprecedented level of understanding and interaction, setting it apart from existing AI models.
A key aspect of this project is the development of a secure repository for data collection. This system will gather chat data throughout your day, creating a comprehensive understanding of your communication style and preferences, for personalized assistance.
The dataset, as we and people who want to contribute build it out will be posted here!: https://huggingface.co/datasets/AlignmentLab-AI/AutometicAssistant
please feel free to submit any data you generate and I will happily clean it, and optimize it for training!
One of the distinctive features planned for AutoMetic Assistant is its ability to be trained on an easy chat format. This format facilitates unique training approaches, allowing the possibility of creating AI clones, and other diverse applications. By exporting your chat data to the model, the assistant will be able to learn, adapt, and mirror your communication style.
The overnight learning feature is another innovative facet of the AutoMetic Assistant. As you sleep, it's intended to perform sentiment analysis on the day's interactions, learning from its performance to enhance its services.
AutoMetic Assistant is designed to be more than a tool; it's intended to be a learning companion that improves with each interaction. Using iterative learning processes and cutting-edge AI techniques, the assistant aims to refine its understanding of user needs and preferences.
Embracing multimodality is the cornerstone of the next AI revolution. We are at the forefront, working on models that not only accept data from any modality but also generate it. Multimodal AI systems can process and interpret various forms of data, such as text, images, audio, and video, to provide a richer and more intuitive user experience.
In collaboration with LAION, we're contributing to the "Open Empathic" project—a leap forward in creating emotionally intelligent AI. This dataset will be instrumental in teaching AI how to understand and generate emotional responses, propelling advancements in empathetic AI applications.
Key Features of the Open Empathic Dataset:
- Detailed annotation of segments with audio, images, and video.
- Coverage of 90 emotional categories.
- Measures of valence and arousal.
- Annotations for gender, estimated age, accent, and background sounds.
- Freeform captions detailing perceived emotions based on audio and video cues.
This dataset is a critical step toward AI systems that can truly resonate with human emotions, enhancing interactions and forging genuine connections.
Our project is currently in its early stages, and we're actively seeking contributors to help bring AutoMetic Assistant to life. We're excited about the potential of this project and would love for you to be a part of it. Whether you're a developer, a data scientist, or just someone with a passion for AI, your skills and input could be invaluable. Join us in creating a new frontier in AI-assisted living.
We welcome donations and collaborations from all interested parties. Whether through GitHub sponsorship, supporting us on Ko-Fi at https://ko-fi.com/alignmentlabai, or by connecting with us directly via email at autometa@alignmentlab.ai or on our Discord at https://discord.gg/SS3xjeqx2M, your contribution can make a significant difference.
Thank you to the researchers, Microsoft, Google, and Facebook for building the foundations which make this possible, drawing on inspiration and elements from the following repositories and studies:
- https://github.com/microsoft/unilm/tree/master/kosmos-2
- https://github.com/yxuansu/PandaGPT
- https://arxiv.org/abs/2109.10686
- https://github.com/NExT-GPT
- https://github.com/Muennighoff/sgpt
- https://huggingface.co/segmind/SSD-1B
- https://huggingface.co/stabilityai/stable-diffusion-xl-base-1.0
- https://huggingface.co/distil-whisper/distil-large-v2
- https://huggingface.co/coqui/XTTS-v2
- https://huggingface.co/suno/bark
- https://huggingface.co/lmsys/vicuna-13b-v1.5-16k