-
Notifications
You must be signed in to change notification settings - Fork 18
Topics in ITP: Contributing to ml5.js (Research)
Welcome! This page contains a list of ideas and how you can help contribute to the ml5.js project as part of ITP's spring 2024 seminar (2 credits). If you are a beginner and keen to dive into the world of machine learning and open source, you're in the right place!
This course is a studio-oriented seminar where students will actively participate in the evolution of the ml5.js open source project. Registration for this course will be conducted via an application process and is limited to 8 students. The structure of this course is inspired by initiatives such as the Processing Foundation Fellowship and Google Summer of Code. Students interested in enrolling are required to submit a detailed project proposal for the semester. We invite proposals that encompass all facets of the ml5.js project, including but not limited to community management, documentation enhancement, website development, pedagogical research, development operations (DevOps), and expansion of machine learning models.
- Application form—deadline November 3rd
- Project Proposal Template
- For inquiries, please contact Daniel Shiffman.
The ml5.js project actively seeks help with the features and initiatives listed, but we also look to applicants to propose ideas we haven't thought of ourselves. For all of our projects, it's incredibly important that things are kept as simple and friendly as possible. ml5.js is not meant to be a comprehensive and technically deep machine learning library for developers. It's for people who are less familiar with code and are new to machine learning. We're far less interested in features to make ml5.js more powerful for advanced users than we are in features that make it easier to handle tasks that are common for a wide range of our audience.
In addition to this list, we track specific bugs and enhancements via github issues:
If you have questions about an idea or need guidance on forming your proposal, please post to the ml5.js Discord, under #general-discussion or #help-with-ml5. This is the best way to get feedback and develop your idea. This makes it possible for everyone to learn from your questions and contribute their thoughts.
If your only experience with coding is ICM or Creative Computing at ITP and IMA, please apply! Your contributions are valued and important. ml5.js is not a product, it's a tool for a community of learners. It's a safe and friendly space for you to learn how to write code for a library, make mistakes, and get support from your peers.
This studio is modeled after the Processing Foundation fellowship program and Google Summer of Code. To get a sense of the kinds of projects from these initiatives, you can read these wrap-up posts.
- Processing Foundation GSOC 2023 wrap-up post
- Previous Processing Foundation articles about GSoC
- Processing Foundation 2023 Fellows
ml5.js is a friendly, high-level interface to machine learning algorithms and models, aimed to make machine learning approachable for a broad audience of artists, creative coders, and students. The goal of the library is not to be comprehensive machine learning tool with low-level capabilities. Rather, the core audience is beginners to coding and complete newcomers to the applications of machine learning in creative coding. The library's materials and examples should follow the spirit and mission of p5.js, prioritizing beginners and access.
- Webiste
- Community and Education
- License and Code of Conduct
- Machine Learning Models Implementation
- Research and Development
- DevOps/Library Maintenance
The website is a critical part of ml5.js. Proposals for improving the ml5.js website's design, accessibility, and content, are welcome and encouraged. Keep in mind the goals of it being a friendly resource for those new to machine learning. Design and features should focus on making the initial learning journey smooth, welcoming, and less daunting for the community.
We are currently redesigning the ml5.js website! There is an existing team dedicated to redesigning the website, but we are also open to additional contributions. If you want to propose ideas around making the interface and design friendly and intuitive, your involvement is more than welcome. Reach out to J.H. Moon for more details.
Have you found any content on the website confusing or inefficiently communicated? Are there any topics missing on the website that you believe should be included? If so, we encourage you to share your ideas and suggestions!
Your contributions should achieve one or more of the following goals:
- simplify the learning curve for ml5.js;
- describe concepts and functionality succinctly;
- specify instructions step by step;
- provide more engaging learning materials for users;
- address essential machine learning topics such as AI Ethics, AI Bias, and Data Security.
We are excited to grow the ml5.js community by making our website more participatory, interactive and engaging. Currently, the main pages of our website are built with Gatsby, using React components. while the model references pages are built with Docsify. If you have suggestions on more engaging, reusable components, we would love to hear the ideas!
Please share your creative concepts that can be used to enhance our main pages and simplify website maintenance. Here are some examples.
If you've ever thought while using ml5.js, "I wish they could elaborate more on this!", we invite you to help us and other learners by creating learning materials.
On the new ml5.js website, you will find a section labeled Learning, which includes a new tab Community Tutorial Library. The tab will collect and showcase learning materials contributed by our ml5.js users.
Your contributions can be a tutorial, documentation, and/or project in various formats, including but not limited to: videos, markdown files, interactive websites, research papers, podcast episodes, games, with these potential focuses:
- ML terminologies or concepts in ml5.js that could be confusing for beginners;
- How to use specific ml5.js models/functionalities that are beginner-friendly and informative;
- How to use ml5.js to build a specific type of creative media applications that demonstrate interaction possibilities (i.e., painting with nose).
We've recently implemented the ml5.js Glossary as a new tab for beginners to learn ml5 terminologies.
You can contribute to the glossary by following these steps:
- Go through the ml5.js website, library, and examples;
- Identify and collect terms that could be included in the glossary;
- Research and draft definitions for the terms;
- You can write the definitions yourself, or include quotes from other sources;
- Your explanation can be enhanced using images, interactive modules, animations;
- Add the defined terms to the glossary.
Community and education involve creating resources, materials, and spaces that foster a supportive and collaborative learning environment. This can include workshops, tutorials, or other educational content that help demystify machine learning concepts and how to use ml5.js effectively. The aim is to cultivate a thriving ecosystem where learning and collaboration flourish for diverse audiences.
- Develop and Teach Workshops: Propose and develop materials for audiences and communities that are new to machine learning and ml5.js;
- Educational Resources: Propose new accessible and inclusive learning materials that cater to different learning styles and levels;
- Organize and Support online communities: What online spaces do people discuss ml5.js and get help and support? Discord? Social Media? How can the community be supported and engaged through existing and new spaces?
In 2021, ml5.js released a Code of Conduct and an accompanying software license. More details can be found in A New Code of Conduct and License for ml5.js. Proposals for evaluating the Code of Conduct and organizing a process for updating and expanding it are welcome. This work should build off of the summer 2023 Open Call for Feedback on ml5.js.
The following models/functions are currently available in 0.12.2 but are not yet ported to this new "next generation" library. Looking for proposals for re-implementing and reviving them!
-
ml5.imageClassifier()- What it does: This function recognizes and labels images.
-
Details: It's like teaching the computer to tell whether a picture is of a cat, a dog, or something else. It primarily uses a model called
MobileNet, and you can also use models created with Teachable Machine, a tool that lets you make your models without deep coding knowledge.
-
ml5.featureExtractor()- What it does: This function turns images into a list of numbers, making them understandable to computers in a specific way.
- Details: Imagine converting a picture into a special code that a computer uses to identify and learn various things from the picture. This code can help in creating custom models, grouping similar images, and more.
-
ml5.objectDetection()- What it does: It finds and labels objects within images.
- Details: Imagine a program that can look at a photo and point out objects like "cat here", "car there", by drawing boxes around them and naming them. This is what object detection does!
-
ml5.soundClassifier()- What it does: Identifies and labels sounds.
- Details: Much like image classification, but for sounds. It could tell you if a sound is applause, a voice, or a musical instrument. It also works well with models made using Teachable Machine, allowing for customized sound recognition.
-
ml5.KNNClassifier()- What it does: Helps in classifying or categorizing data.
-
Details: Think of it as a voting system. When new data comes in, it looks at the closest known data points and lets them 'vote' to decide how to categorize the new data. It’s often used together with the
ml5.featureExtractor()to classify images, sounds, and other data.
While the goal of ml5.js "next generation" is to be smaller, more lightweight and focused on beginners, we are also interested in new models that demonstrate core ML capabilities and are well suited for applications in creative arts and media.
-
Image Segmentation: ml5.js currently has the image segmentation model
BodyPix, but what other kinds of segmentation models might we consider? [Here's an example of a Segment Anything model that runs in the browser. -
Object Detection: ml5.js supports the
coco-ssdandyolomodels for object detection (see above listing for step 1 of re-implementing object detection in ml5 "next gen"). What newer models are available and how might one train a custom object detection model (transfer learning a la Teachable Machine?) for use in ml5.js/ - Clustering: ml5.js currently supports KNN classification (see above listing for step 1 of re-implementing object detection in ml5 "next gen"). What would it mean for ml5.js to have more general purpose functionality (KNN or otherwise) for finding relationships between data. This relates to the "research and development" question below around embeddings.
This section involves exploring new technologies, models, and techniques that could enhance ml5.js's capabilities. Keep in mind the goal of keeping ml5.js small in scope, easy to maintain, and prioritizing beginner audiences. ml5.js also has a goal of all models running "on device" meaning in the browser itself and not making use of commercial cloud-based services and APIs.
- Generative Image Models: Investigate the feasability of integrating "toy" diffusion (or the kinds of generative models) into ml5.js? Consider the ml5.js Code of Conduct and License as it relates to the data behind the training of these models and their applications.
- Language Models: Investigate the integration of language models into ml5.js. What can we do with "toy" language models running the browser. Some explorations might involve transformers.js. Consider the ml5.js Code of Conduct and License as it relates to the data behind the training of these models and their applications.
-
Embeddings: Related to language models and
ml5.featureExtractor(), what other browser-compatible models generate embeddings for text and other media? Examples and applications could be clustering, text classification, -
Sequential Data: Prior versions of ml5.js supported LSTMs and other RNNs (like
SketchRNN). What might be worth including in ml5.js to demonstarte how to work with sequential data such as time-series data, mouse movements, and more? - Explore onnx.js as possible alternative backend for ml5.js.
- Explore candle (plus WASM?) as possible alternative backend for ml5.js.
- Unit Tests: Develop unit tests for new and existing features.
- Developer Documentation Materials: Create comprehensive and user-friendly documentation for new contributors.