Discussion: Overlay Brainstorms

[Oceanswave]

Wanted to open up a discussion about other possible overlays or use of the technique for other applications. Aside the fun of having bugs or other effects on screen, having full-screen, non-intrusive overlay capability is pretty powerful...

Here are some thoughts 💭

• Redact-ly: black boxes in predefined locations (useful when doing live webcasts where the customer is sensitive about their brand or data contained in the app)
• Picture-in-picture: Displays videos in the corner from youtube/Netflix/PluralSight (Also useful to display the laptop’s webcam to show you when someone comes up behind you) (Take that macOS Sierra!)
• CPU/System/Audio monitor (so you can see the bass being dropped.)
• Class of overlays that continuously takes screenshots (either full-screen, window, or region), processes the image somehow and overlays changes or other information.
  • Real-time diff? Highlights regions that have changed with a n-second delay.
  • Super-Chromatic Peril-Sensitive Overlay? Detects any nudity displayed via https://www.patrick-wied.at/static/nudejs/ or similar and if detected, blanks out the screen? ("Oh, let me have a little bit of peril" - Sir Galahad)

[jimbuck]

Excellent ideas! And I'm all for making this more than just a joke app (name will have to change then). The main hurdles will be what can Node/Electron handle (real-time video diff would be quite the test) as well as how to make simple, unified UI for settings that covers the majority of use cases. Ideally this would have a plugin architecture, so it's just a matter of dropping a zip into the overlays folder and it would "just work"®. Maybe the settings for each overlay opens a new window (and temporarily disables the overlay). I guess there just has to be a set standard for how overlays are defined (overlay.json, see #5 for details). Additionally, keyboard shortcuts will be required, so that you can interact with the overlay.

Other Ideas:

• Twitter updates scrolling across the screen (retweets, likes, mentions, etc.)
• Live weather every n minutes (rain when rainy, snow when cold, lense flare when sunny, etc)
• Pomodoro timer/timecard/todo list helper of some sort.

[Oceanswave]

Had some time so looked briefly into some feasibility to being able to capture the screen:

While it's doable on a snapshot-by-snapshot with robotjs, there's also the possibility of using the mediaDevices.getUserMedia() api which is described here, Demoed here and mentioned here as possible in electron.

It seems to emit a mp4 stream, which would be interesting to pipe into ffmpeg to do live processing.

Although you could just bypass the getUserMedia method and use ffmpeg directly...

...still lies the problem of how to process that in near-realtime to detect features and overlay... Snapchat for your desktop... ;)