ULTRON — Python Voice Assistant

A JARVIS/Ultron-style voice assistant you run on your own computer. It listens to your microphone, understands commands, performs actions, and talks back — all from a clean, modular Python codebase you can extend.

Unlike a browser page, this runs natively, so it is not blocked by Brave or any browser privacy setting.

What it can do

• Tell the time and date
• Open desktop apps ("open notepad", "open calculator", "open camera")
• Open websites ("open youtube", "open gmail", "open github")
• Play songs on YouTube ("play interstellar theme") and search the web ("search for iron man")
• Read the news ("what's the news")
• Take notes ("take a note buy milk") and read them back ("read my notes")
• Reminders & timers ("remind me to call mom in 5 minutes", "set a timer for 2 minutes")
• Quick maths ("what is 25 times 4")
• Jokes, greetings, and simple chit-chat
• Wikipedia answers ("who is Nikola Tesla")
• Weather ("weather in Mumbai") — with a free OpenWeatherMap key
• Where am I / live location ("where am I", "show my location on the map") — precise GPS on Windows, free approximate lookup everywhere else
• Track others, with consent ("track dad using ", "where is dad", "where is my car") — opens a shared live-location link, or reads a device's live GPS feed
• Any question at all — with a free Groq API key (recommended), or Gemini/OpenAI

If no microphone is available, it automatically switches to typed input, so it always works.

Project structure

ultron-assistant/
├── main.py                 # entry point  ->  python main.py
├── requirements.txt        # dependencies
├── .env.example            # copy to .env to change settings (all optional)
└── ultron/
    ├── config.py           # typed settings loaded from .env
    ├── logger.py           # console + rotating file logging
    ├── assistant.py        # the run loop (listen -> think -> speak)
    ├── speech/
    │   ├── listener.py     # microphone speech-to-text
    │   └── speaker.py      # offline text-to-speech
    └── brain/
        ├── router.py       # decides skill vs AI vs fallback
        ├── skills.py       # all built-in commands
        ├── location.py     # GPS + IP lookup, distance & geocoding
        ├── targets.py      # people/devices you may track (consent)
        ├── places.py       # named places (home, office) for alerts
        ├── watch.py        # background arrival / departure alerts
        └── ai.py           # optional OpenAI backend
    ├── ui/orb.py           # animated orb window (scales to hand gestures)
    └── vision/hands.py     # optional webcam hand tracking

Setup (Windows + VS Code)

1. Open the folder in VS Code (File → Open Folder → ultron-assistant).
2. Open a terminal (Terminal → New Terminal) and create a virtual environment:

   python -m venv .venv
   .venv\Scripts\activate
   

3. Install the dependencies:

   pip install -r requirements.txt
   

   If PyAudio fails to install on Windows, run pip install pipwin then pipwin install pyaudio, and re-run the command above. The assistant will still start without it — it just falls back to typed input.

4. Run it:

   python main.py
   

5. Speak (or type) a command. Say "exit" to quit.

On macOS/Linux the steps are the same except activate with source .venv/bin/activate.

Settings (optional)

Copy .env.example to .env and edit any line. Everything is optional:

• USE_VOICE_INPUT=false — type commands instead of speaking
• ASSISTANT_NAME, USER_NAME — personalise the greeting
• GROQ_API_KEY — free AI brain: answer ANY question (get a key with no credit card at https://console.groq.com). GEMINI_API_KEY also works free; OPENAI_API_KEY works too but is paid. Add just one.
• WEATHER_API_KEY — enable live weather

Where am I — location & GPS

Say "where am I", "what's my current location", or "show my live location on the map" and ULTRON reports where you are. Add "map", "live", "track", or "gps" to the request and it also drops a pin on Google Maps.

It picks the most accurate source available, with no setup required:

• Precise (Windows only): real GPS / Wi-Fi positioning through Windows Location Services, accurate to a few metres. This needs the optional winsdk package (already in requirements.txt) and location access turned on in Settings → Privacy & security → Location. The first time, Windows may ask you to allow it.
• Approximate (every OS): if precise location isn't available, ULTRON falls back to a free, no-key lookup that maps your internet connection to a city. This is roughly city/neighbourhood accurate, not street-level, and it clearly says so when it answers.

Everything runs over free public services — there is no API key to add and nothing is stored.

Tracking others (with consent)

ULTRON can also locate a person or device that has opted in — a family member, your own other devices, or a vehicle. It cannot find anyone secretly; that isn't technically possible and this project won't fake it.

Two ways, both driven by the other side sharing:

• A shared link (easiest): they share their live location from Google Maps or WhatsApp and send you the link. Tell ULTRON once — track dad using https://maps.app.goo.gl/... — and then "where is dad" opens his live map.
• A live feed: run companion/report_location.py on the device you want to track (your laptop, or a mini-PC in a car). It publishes that device's GPS to a small feed ULTRON reads, so "where is my car" speaks the actual location.

Full step-by-step setup — including turning an Android phone into a GPS tracker — is in companion/README.md.

Distances & arrival alerts

Once a target has a live feed (not just a shared link), ULTRON can measure and watch it:

• Distance: "how far is my car", "distance to home" — it answers with the distance from where you are and the rough direction ("about 4.2 kilometres away, to the north-east").
• Arrival / departure alerts: "alert me when dad arrives at home", "tell me when I leave the office", "notify me when the car reaches school". ULTRON watches quietly in the background and speaks up when it happens.
• Recurring, restart-proof alerts: add "every day" or "whenever" — "every day tell me when Aanya reaches school" — and the alert re-fires each time and is saved to disk, so it's still watching after you restart ULTRON.
• Manage them: "what alerts do I have" lists what's active; "stop alerts" cancels everything.

Places like home and office are just saved coordinates. Save one by voice — "set home to 12 Mall Road, Ludhiana" — or list them in .env under PLACES. You can also name any real address or landmark directly in a command and ULTRON will look it up (free, via OpenStreetMap).

For step-by-step setup by situation — a child, a car, a lost device, or a partner/friend — see companion/USE-CASES.md.

Hand gestures (optional)

Move your hand in front of the webcam and the orb grows and shrinks with it — spread your fingers or bring your hand closer to max it out, make a fist or pull your hand back to min it. A small "HAND CONTROL" bar appears while your hand is driving it.

Turn it on by installing the two extra libraries and running as usual:

pip install -r requirements-gestures.txt
python main.py

It needs a webcam and Python 3.9–3.12. If the libraries or a camera aren't present, the orb just runs normally — nothing else changes. Set GESTURE_MODE=false in .env to disable it, or CAMERA_INDEX to pick a different camera.

Fist = small, open hand = big. With MediaPipe working, the orb follows how open your hand is. If MediaPipe won't load (a common "no attribute 'solutions'" error, caused by NumPy 2), fix it once with:

pip install "numpy<2" mediapipe --force-reinstall

Without MediaPipe the orb still resizes, but from hand movement only (OpenCV fallback) — the exact fist/open poses need MediaPipe.

Control your PC with gestures (Iron-Man mode)

Turn on GESTURE_CONTROL=true in .env to let gestures do things. Hold a pose for about a second to trigger it (so passing poses don't fire):

Gesture	Action
point (1 finger)	open your app (GESTURE_APP, default Notepad)
peace (2 fingers)	switch window (Alt+Tab)
three fingers	show / hide desktop (Win+D)
thumb up	play / pause media

Hold the pose steadily for about a second. When it fires you'll see a » ACTION « banner flash under the orb, so you get instant confirmation. Fist and open hand only size the orb — they never trigger a command, so shrinking the orb won't set anything off.

This needs pyautogui (included in requirements-gestures.txt) and a working MediaPipe. It's off by default so nothing happens by accident.

Extending it

Add a new command by writing one small function in ultron/brain/skills.py that returns a response string (or None if it does not apply), then add it to the ALL_SKILLS list. That is the whole extension model.

License

MIT — do whatever you like with it.