Skip to content

[DMP 2025]: Color sensor for Music Blocks for photos and real-time video #4537

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
5 tasks
pikurasa opened this issue Mar 15, 2025 · 35 comments
Open
5 tasks
Labels

Comments

@pikurasa
Copy link
Collaborator

Ticket Contents

Description

Music Blocks has a feature to detect the color of pixels generated
from drawing within the program, but it cannot detect the color of
pixels from images that are either uploaded or from a webcam. By
adding a feature to detect color from both uploaded images and a live
webcam stream, users would be able to implement Lego music notation
for the blind
and similarly interactive programs.

The goal of the project is to develop extended functionality to our
existing tools of turtle/mouse glyph movement and limited color
detection to sense color from uploaded images, as well as the
real-time feed from a webcam. Upon successful implementation, the
turtle/mouse glyph will be able to detect the color of pixels
underneath it, regardless of whether those pixels were drawn by the
turtle/mouse itself, part of an uploaded image stamped to the canvas,
or part of a live webcam video feed into Music Blocks. One test of
success is to run our Lego music notation for the blind project with
a live feed. The result should be able to playback and record the
abstract brick notation based on its contrasting colors.

Goals & Mid-Point Milestone

Goals

  • [Mockup a functional prototype for detecting the color of pixels from a live feed]
  • [Integrate the prototype within Music Blocks visual programming language]
  • [Work together with classroom partners to test project]
  • [Make further improvements, based on results of classroom testing]
  • [Document code and its functionality]

Setup/Installation

No response

Expected Outcome

Upon successful implementation, users should be able to use a live feed within Music Blocks and use the "pixel color" block to detect the color of pixels from a live webcam feed. An important use case for us is to be able to use this feature to encode musical data via LEGO bricks, which promises to make Music Blocks accessible to the blind.

Acceptance Criteria

No response

Implementation Details

Ideally, the pixel color feature should be implemented in client-side Javascript, the same language Music Blocks is written. Implementations done via server-side Javascript or Python are acceptable as well, but less ideal.

Mockups/Wireframes

References:

Product Name

Music Blocks

Organisation Name

Sugar Labs

Domain

⁠Education

Tech Skills Needed

JavaScript

Mentor(s)

Coding Mentors

Walter Bender

Assisting Mentors

Devin Ulibarri

Category

Accessibility

@Abhijitam01
Copy link

i understand what you are trying to say. I think i can implement this. Can you assign this to me ?

@pikurasa
Copy link
Collaborator Author

i understand what you are trying to say. I think i can implement this. Can you assign this to me ?

This is reserved for a DMP mentee. Please look into how to apply for DMP: https://www.codeforgovtech.in/ (which seems to be down atm, so use https://web.archive.org/web/20250216103313/https://codeforgovtech.in/) and https://discord.gg/aFWG6CwP

In the meantime, I recommend that you join the conversation on https://matrix.to/#/#sugar:matrix.org and https://matrix.to/#/#musicblocksdev:matrix.org

@Abhijitam01
Copy link

got it . Thanks

@sai-charan1
Copy link
Contributor

Hello @pikurasa

I came across this exciting initiative to integrate an AI-powered debugger into Music Blocks, and I’d love to contribute. With my experience in AI, NLP, and full-stack development, I have worked on fine-tuning open-source LLMs, deploying AI-powered applications, and integrating AI solutions into platforms.

Could you please specify the further steps...?

@Vanshika110
Copy link

I would like to work on this project, what are the key steps for this, to get selected in this project.

@Kiranmayi-45
Copy link

Hi! I’d love to work on this issue for DMP 2025. I’ve already explored how to detect pixel colors using canvas for both uploaded images and live webcam feeds using JavaScript.
I’ll start by creating a prototype and then work on integrating it into Music Blocks. Looking forward to your guidance!

@alokdangre
Copy link
Contributor

Hi @pikurasa
,and @walterbender
I’d like to propose a solution to extend Music Blocks’ color detection capabilities. The plan is to use an off-screen canvas to unify color detection for drawn content, uploaded images, and live webcam feeds. This will enable our turtle/mouse glyph to reliably sense pixel colors from any source, enhancing projects like the Lego music notation for the blind.
can you plzz share views on it

@Chandni2003
Copy link

Hello @pikurasa
,and @walterbender

Unlocking Music Blocks' Full Potential

I'm Chandni Gupta, and I'm thrilled to throw my hat into the ring for the Music Blocks project. With my passion for accessibility and ed-tech, I believe I can bring a unique perspective to this project.

My Superpowers:

  • JavaScript wizardry
  • React and DOM manipulation expertise
  • Team player with a knack for learning quickly
  • Passionate about making tech more accessible

The Goal:

  • Develop a feature that detects pixel colors from live webcam feeds
  • Seamlessly integrate it with Music Blocks
  • Test and refine it with classroom partners

Why I'm the Perfect Fit:

I've got the skills, experience, and passion to make this project shine. I'm confident that my expertise in JavaScript and React will enable me to deliver high-quality results. Plus, I'm excited about the prospect of working on a project that can make a real difference in people's lives.

@Richajaishwal0
Copy link

Hello @pikurasa

I'm very excited about this feature request — it aligns perfectly with both my technical skills and my enthusiasm for building inclusive educational tools. The idea of integrating a color sensor system into Music Blocks, especially with the goal of helping visually impaired users interact with and create music through tangible tools like LEGO bricks, is truly inspiring.

From what I understand, the implementation would involve:
- Extracting RGB values from uploaded images or live webcam feeds.
- Linking these color values to corresponding musical blocks or tones.
- Possibly building a library of "color tones" to map bricks/pixels to sounds.
- Expanding Music Blocks' accessibility and creative learning potential.
With my background in JavaScript, webcam integration, and visual programming logic, I’m confident that I can contribute effectively to this feature. I’ve also worked on projects involving real-time data capture and visual feedback, which I believe directly aligns with the kind of work this feature demands.

Moreover, I really appreciate Music Blocks’ mission of making music education engaging and accessible through a visual programming language. Enhancing this with sensory input like color detection feels like a natural next step, and I’d love to be a part of that journey.

Kindly consider assigning this to me — I’d be thrilled to explore ideas, collaborate, and start building

@Rajuttam15
Copy link

Hello @pikurasa
Hi! I’m interested in working on this issue as a contributor.

From what I understand, the goal is to enable colour detection from both webcam streams and uploaded images, so that the turtle/mouse glyph in Music Blocks can read pixel colours—not just from the drawing canvas but also from external visual sources. This would support use cases like Lego music notation for the blind.

I’ve experimented with detecting pixel colour from webcam feeds using the canvas API (getImageData) in vanilla JS, and I’d love to explore integrating it into Music Blocks.

@Khushijoshi003
Copy link

Hello @pikurasa
Sir, I just wanted to say how amazing Music Blocks is! The concept of enabling kids to create music visually is both creative and inclusive.
I’m currently working on a feature idea for the project that I believe aligns with your vision — adding color detection from uploaded images and live webcam feeds, so that kids (especially blind/visually impaired) can play LEGO-based music using color-coded bricks.
I’ve started setting this up locally and diving into the code. Would it be okay if I propose a PR or share my approach with you for feedback? I’d love to collaborate, learn, and contribute in a way that supports the community you're building.
Thank you so much for such an inspiring project!

@tarunkumar2005
Copy link

Hello @pikurasa, I'm Tarun and I'm a good fit for working on this project because of my skills in python, gen ai and working with LLMs. I have worked on several projects with similar implementation, you can check our in my resume or my LinkedIn profile. This will be a very good opportunity for me and i will be very helpful asset to the project. Please reach our to me if you have any doubts or want clarification on me or my projects. Looking forward to contributing to the project.

Linkedin: https://linkedin.com/in/tarunkumar8278

@Ashi1411
Copy link

Subject: Interest in Contributing to Music Blocks' Color Detection Feature​

Hello @walterbender and @pikurasa ,

I hope this message finds you well. My name is Ashi Jain, and I am enthusiastic about contributing to the Music Blocks project under the Code4GovTech initiative.

I am particularly inspired by your initiative to enhance accessibility in Music Blocks by enabling color detection from uploaded images and live webcam feeds. This feature, aimed at supporting LEGO music notation for blind users, is a commendable step towards inclusive education. The prospect of translating visual elements into auditory experiences resonates deeply with my passion for leveraging technology to bridge accessibility gaps.

With a strong foundation in JavaScript and experience in client-side development, I am confident in my ability to contribute effectively to this project. I have previously worked on projects involving real-time video processing and color detection, which aligns well with the technical requirements of this enhancement.

I am eager to collaborate with your team to develop and integrate the extended color detection functionality. I would appreciate the opportunity to discuss how I can best support this project and align my efforts with the team's goals.

Thank you for considering my application. I look forward to the possibility of contributing to Music Blocks and learning from your guidance.

@shikjazz
Copy link

@pikurasa This is my expertise.. Could you please assign this project to me?
Regards

@jettylakshmisowjanya
Copy link

Hi there,

I'd like to contribute by working on a solution to extend Music Blocks’ color detection capabilities. I have a background in JavaScript and would be happy to help with this task. Let me know if it's okay to proceed.

Thanks!

@Pitta-Abhiram6
Copy link

Hello @walterbender and @pikurasa

I am Abhiram from IIT Guwahati .I gone through project Issue and I made a research for color detection from
pixels from images that are either uploaded or from a webcam. By adding a feature to detect color from both uploaded images and live web stream.
I have very good grip in JavaScript and also React too . After I made research what I came to know that to six the issue

1.first thing is to extract the RGB values from uploaded images or live webcam feeds.
2. blend those RGB with corresponding music blocks .
3. testing and refining
Since I have Advanced grip in the JavaScript I can able to fix the issue in time and efficiently . I given opensource contribution in my past so this environment is familiar to me not only that i am capable enough for this project .

I am eagerly waiting to collaborate with your team to work on this project and get guidance under excellent tech professionals Using this opportunity I will build my skills in JavaScript a bit more and meet new highly professionals.

Eagerly waiting to work in this project hope it happen for sure .
Thank you sir.

@Pitta-Abhiram6
Copy link

could you please assign this project to me it will be grateful to showcase my skills

@himudit
Copy link

himudit commented Apr 21, 2025

Hi @walterbender @devinulibarri,

I hope you're doing well! My name is Mudit, and I’m incredibly excited about the possibility of contributing to the Music Blocks – Pixel Color Detection project as part of DMP 2025.

This project’s vision—enhancing accessibility by enabling pixel color detection from uploaded images and live webcam feeds—deeply resonates with me. I believe this functionality has great potential to make Lego Music Notation for the blind even more impactful and inclusive.

I have a solid foundation in JavaScript, and I’ve explored the Music Blocks platform to get hands-on familiarity with its interface and programming style. I’m particularly interested in developing the client-side color detection logic, and I’d be thrilled to help integrate this into the turtle/mouse glyph system for both uploaded and live video content.

As a suggestion, to improve real-time color detection and minimize flickering or false reads, we could consider implementing a sampling threshold mechanism or an averaging window around the cursor to stabilize the readings from the live feed—especially in low-light or noisy conditions.

Would it be possible to connect briefly and understand the current status of this feature or any available prototypes? I’d love to align with your roadmap and begin contributing in a meaningful way.

Looking forward to collaborating with you and the team!

Warm regards,
Mudit

@bhumika-ks31
Copy link

Hello @pikurasa
,and @walterbender

Unlocking Music Blocks' Full Potential

I'm bhumika kashyap , and I'm thrilled to throw my hat into the ring for the Music Blocks project. With my passion for accessibility , I believe I can bring a unique perspective to this project.

My Superpowers:

Html css JavaScript React node js mongo db my sql andmake api calls and expertiseTeam player with a knack for learning quicklyPassionate about making tech more accessible

The Goal:
To enhance Music Blocks by enabling pixel color detection from uploaded images and live webcam feeds, allowing the turtle/mouse glyph to interact with all visual inputs. Additionally, integrate an AI chatbot assistant to guide users, especially visually impaired users, in using the platform and accessing features like Lego music notation.

Why I'm the Perfect Fit:

I've got the skills, experience, and passion to make this project shine. I'm confident that my expertise in html css JavaScript and mongo db node js will enable me to deliver high-quality results. Plus, I'm excited about the prospect of working on a project that can make a real difference in people's lives.

Looking forward to collaborating with you and the team!
Github ..https://github.com/bhumika-ks31
Warm regards,
Bhumika

@bhumika-ks31
Copy link

Hello @pikurasa
,and @walterbender

Unlocking Music Blocks' Full Potential

I'm bhumika kashyap , and I'm thrilled to throw my hat into the ring for the Music Blocks project. With my passion for accessibility , I believe I can bring a unique perspective to this project.

My Superpowers:

Html css JavaScript React node js mongo db my sql andmake api calls and expertiseTeam player with a knack for learning quicklyPassionate about making tech more accessible

The Goal:
To enhance Music Blocks by enabling pixel color detection from uploaded images and live webcam feeds, allowing the turtle/mouse glyph to interact with all visual inputs. Additionally, integrate an AI chatbot assistant to guide users, especially visually impaired users, in using the platform and accessing features like Lego music notation.

Enhance Color Detection: Support for Uploaded Images and Webcam Feed
Fixes issue #4537

This pull request adds functionality to enhance the color detection system in Music Blocks by:

  • Enabling users to upload images and extract color data using the HTML5 File API and canvas.
  • Accessing the webcam using getUserMedia and processing real-time video frames for color detection.
  • Integrating this color data with the existing Music Blocks system, allowing interactive usage like music mapping.

Tested with:

  • Various image uploads
  • Live webcam stream

Everything works smoothly as expected. Looking forward to your feedback!
Regard
Bhumika kashyap
Github ..https://github.com/bhumika-ks31

@pikurasa
Copy link
Collaborator Author

could you please assign this project to me it will be grateful to showcase my skills

This is a special issue. It's for https://codeforgovtech.in/dedicated_mentoring_program/. We encourage interested contributors to apply there.

Also, we recommend that you join the conversion on https://matrix.to/#/#musicblocksdev:matrix.org

@kuldeepkarma
Copy link

Could you please assign this project to me it will be grateful to showcase my skills in this project

@tiwari14-maker
Copy link

tiwari14-maker commented Apr 28, 2025

@walterbender @pikurasa

About Me :
Hello!
I am an experienced frontend developer with strong expertise in JavaScript and interactive web applications. I have built various projects involving canvas manipulation, real-time data handling, and visual programming concepts. I am passionate about building inclusive and accessible digital tools, which aligns closely with the mission of Music Blocks and Sugar Labs.

Why Me :
Strong experience with HTML5 Canvas and real-time media handling in JavaScript.

Familiarity with pixel-level color detection, image processing, and webcam stream integration.

Deep understanding of client-side development, ensuring smooth performance without server dependencies.

Focus on building accessible and user-friendly tools for educational purposes.

Excited to contribute to open-source educational technology and work collaboratively with classroom partners for real-world testing and feedback.

Thank you for considering my application for this meaningful project.
Best regards,
Shubhangi Tiwari

@Mohammadshiraz
Copy link

Hi @walterbender @DevinUlibarri 👋,

I'm really excited about the opportunity to contribute to the Music Blocks project—especially this feature that enhances accessibility through live pixel color detection. With my experience in JavaScript and image processing, I’d love to help implement color detection for both uploaded images and webcam feeds, and integrate it seamlessly into the existing pixel color block system.

This project resonates with me because of its creative blend of music, education, and accessibility, and I’m eager to collaborate and iterate based on classroom testing results. Please consider assigning me to this issue—I’d be thrilled to get started!

Thanks,
Mohammad Shiraz

@sajjannnn
Copy link

Hi Walter and Devin,

I saw the Music Blocks project about detecting pixel color from a live feed, and it really caught my interest — especially how it can help create accessible tools like Lego music notation for the blind. I’d love to be part of it.

I’m still new to JavaScript, but I’ve started learning it and am ready to put in the work to get up to speed quickly. Would it be okay if I apply while continuing to build my skills along the way?

Thanks for your time, and I’d really appreciate the chance to learn under your guidance!

Best,
Sajjan

@JaySoni1
Copy link
Contributor

JaySoni1 commented May 3, 2025

Hi @walterbender @DevinUlibarri ,
I am excited about the color sensor feature for Music Blocks! With my experience in JavaScript, I am confident that I can help in implement color detection from both uploaded images and webcam feeds. I would love to contribute to this project and collaborate with the team to enhance Music Blocks’ accessibility and creativity.

Thank You,
Jay Soni

@svlalli
Copy link

svlalli commented May 3, 2025

I’m really interested in your project to add color detection from uploaded images and webcam feeds in Music Blocks. I think it’s a great idea, especially for making things like Lego music notation for the blind more interactive and useful.

I have some experience with working on images and webcam inputs, and I’d love to help build this feature so the turtle/mouse can detect colors from anywhere on the canvas—not just what it draws.

I'd be excited to be part of this project and help improve Music Blocks in a meaningful way.

@nitishgithubrit
Copy link

Hi! I’d love to work on this issue for DMP 2025. I’ve already explored how to detect pixel colors using canvas for both uploaded images and live webcam feeds using JavaScript.
I’ll start by creating a prototype and then work on integrating it into Music Blocks. Looking forward to your guidance!

@Ghanshyambunkar
Copy link

Subject: Interest in Contributing to [Project Name]

Hi @pikurasa ,

I’m Ghanshyam, a final-year B.Tech CSE student from Stani Memorial College of Engineering and Technology. I’m excited about the opportunity to contribute to this project, particularly because of its focus on Color sensor for Music Blocks. The real-world impact it could have—especially in underserved communities—aligns closely with my passion for building meaningful applications.

I’ve explored the problem space and have begun brainstorming solutions, including leveraging MERN stack for real-time progress tracking or integrating interactive UI/UX to boost engagement. With experience in full-stack development (MERN), I’ve built projects like:

Wandarlust: An Airbnb-inspired platform with user auth, dynamic search, and booking systems.

E-commerce App: A scalable product with payment integration and admin dashboards.

Simon Says Game: A lightweight interactive game showcasing frontend logic.

I’m confident I can contribute to making this platform robust and user-centric. Is there a Slack/Discord space for collaborators? I’d love to join discussions and refine my approach.

Looking forward to your thoughts!

Best,
Ghanshyam

@nitishgithubrit
Copy link

Hi @walterbender @DevinUlibarri ,
I am excited about the color sensor feature for Music Blocks! With my experience in JavaScript, I am confident that I can help in implement color detection from both uploaded images and webcam feeds. I would love to contribute to this project and collaborate with the team to enhance Music Blocks’ accessibility and creativity.

Thank You,
Nitish Kumar

@Rohan11P
Copy link

Rohan11P commented May 5, 2025

Hi @walterbender @DevinUlibarri ,
My name is Rohan Padhy, and I'm very interested in contributing to Music Blocks through the C4GT 2025 program.I came across Issue #4537, titled "Colour sensor for Music Blocks for photos and real-time video", and I’m very excited about the potential of integrating image and video-based colour detection to create music in an educational and accessible way. I have experience with HTML, CSS, and JavaScript, and I’m confident in working with web API like getUserMedia() and the Canvas API for image and video processing. I did a project on a voice-to-text converter that was a team project, but I contributed a lot to that project. I used the Google Voice to text converter API, and that was a great project.

@vishalm342
Copy link

Hi @walterbender @DevinUlibarri 👋,

I'm genuinely excited about the opportunity to contribute to the Music Blocks project, especially this feature that brings together accessibility, creativity, and music. The idea of enabling color detection from webcam feeds and uploaded images to support Lego music notation for the blind really resonates with me—it’s meaningful, impactful, and something I’d be proud to contribute to.

With my background in JavaScript and frontend development, I believe I can be helpful in implementing and refining this feature. I'm enthusiastic about working with the team, learning through the process, and ensuring the feature is as accessible and user-friendly as possible.

I’d love to be considered for this project and am eager to get started if selected. Thank you for the opportunity!

Best regards,
Vishal

@bhaskarraorajagiri
Copy link

Hi @walterbender @pikurasa
I would like to work on the Color Detection feature for Music Blocks. This project is exciting because it helps users interact with images and webcam feeds using color detection, which is useful for tools like Lego music notation for the blind.

I have experience with JavaScript, and I want to build a feature that allows the turtle/mouse to detect colors from uploaded images and live webcam streams.

@SumedhPawar123
Copy link

I would like to work on this project, Currently i am pursuing Engineering degree in IT, I am a final year student and recently i have done my project which is Design the chat Application using Mern Stack, and i am very excited to work on this project so what are the key steps and how can i selected for this project.

@anushkadasgupta
Copy link

Hi! 👋
I'm Anushka, and I'm applying for the C4GT 2025 Dedicated Mentoring Program. I’ve gone through the project details and I’m really excited about contributing to this initiative. I’ve submitted my proposal titled “Enhanced Color Detection for Music Blocks”, and I’m looking forward to the opportunity to learn and collaborate through this project. Please let me know if there’s any feedback or suggestions!

Thanks and regards,
Anushka Dasgupta

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests