Implement Loom.com integration

[abrichr]

How can we make it extremely simple for people to create and share use cases?

• Loom: https://support.loom.com/hc/en-us/articles/360002228458-Does-Loom-have-an-open-API-
• others?

Capture user narration via speech to text e.g. https://github.com/openai/whisper

[abrichr]

@micol-altomare you might enjoy this one 😄

[dianzrong]

Solution 1: Record videos with Loom

Benefits:

• users don't need to create an account or download anything else to create recordings
• recordings can be easily shared using a url

Implementation:

• starter code is given on Loom but is written in React, Typescript or JavaScript so would also need some Python backend

Limitations:

• under the free account, guest accounts are restricted to a maximum number of 5 videos, and maximum duration of 5 minutes

[dianzrong]

Solution 2: using a Python library

Benefits:

• doesn't require another language

Implementation:

• most commonly, PyAutoGUI is used to take screenshots of the user's screen and OpenCV is used to take video from the user's camera

Limitations:

• needs the user to run the program in the command-line
• since PyAutoGUI is taking screenshots and it has a very slow frame rate, the video quality is very low

[dianzrong]

Solution 3: using Ziggeo API

Benefits:

• has a Python SDK
• has built-in audio transcription capabilities
• can easily upload to youtube
• can put our own branding on it

Implementation:

• use their Python SDK

Limitations:

• may not be able to control the quality of the video (if the default/WebRTC recording method is used)

[dianzrong]

Can also use whisper in combination with the above solutions to communicate use-cases without having to demonstrate it

Benefit:

• more accessible (for those who can't or find typing/clicking painful)
• also more human-friendly

Limitation:

• can be hard to line up audio and actions

[abrichr]

Thanks for taking the initiative on this @dianzrong !

Solution 1: Record videos with Loom

I think this is preferable since Loom is the most popular and easy to use screen casting utility AFAIK. Here are some sources that might be helpful:

https://github.com/Ecno92/loom-dl
ytdl-org/youtube-dl#27957

Solution 2: using a Python library

We considered using PyAutoGUI to implement OpenAdapt, but ended up implementing it from scratch for various reasons. I think we would not want to go this route unless there is a compelling reason.

Solution 3: using Ziggeo API

This could be interesting, I haven't come across Ziggeo before. A glance at their pricing suggests they don't have a free tier however, which makes this less attractive.

Other options:

https://github.com/jonnekaunisto/simple-youtube-api

Uploading to YouTube might be the easiest to start. Eventually I think we also want to support Loom, but their docs suggest this can only be done from a web application. If that's the case then https://nicegui.io/ may also be relevant. It should be possible to interact with their HTTP API directly, e.g. using Python's requests library.

@dianzrong how comfortable are you with web development? Which approach sounds most interesting?

[dianzrong]

@abrichr I do not have experience with web development but would love to spend time on it. Also I feel like I might as well get started with Loom since we want it later. For now I think it would be fastest to get something working using niceGUI so I will start with that!

[abrichr]

Sounds good! If we go with NiceGUI we will likely need to run custom JavaScript: https://nicegui.io/documentation#run_javascript See if you can get their JavaScript examples running from here: https://dev.loom.com/docs/record-sdk/getting-started It may be necessary to build a full stack application for this, but ideally I think we would use NiceGUI.

…

On Mon, May 15, 2023 at 1:28 PM dianzrong ***@***.***> wrote: @abrichr <https://github.com/abrichr> I do not have experience with web development but would love to spend time on it. Also I feel like I might as well get started with Loom since we want it later. For now I think it would be fastest to get something working using niceGUI so I will start with that! — Reply to this email directly, view it on GitHub <#57 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAF5DV3HDVMZCUE45OQ6IDTXGJRU3ANCNFSM6AAAAAAXR5U3TQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>