Home

Everything Jarvis

📓   Wiki updated on May 28, 2021

I created a virtual bot to act like Jarvis. With the current state of things, Jarvis can recognize conversations and respond accordingly. Jarvis uses specific keywords to trigger the respective function which performs a specific task. It is not 100% fool proof but most of the common errors are caught and responses are configured accordingly.

Features:

Jarvis works by automatically detecting the Operating System it is being run, so all you have to do is to run the right installs file following the instructions. Some of the key features require API keys but they can be generated for free. Currently Jarvis uses icloud API to track the current location but will soon be replaced with a more handy option.

• Jarvis is programmed to perform self-recoveries in case of most common exceptions. This includes
  • RecursionError - during repeated function calls. Note::By default the recursionlimit() is set to 10 times the default limit.
  • Fatal Python error::Cannot recover from stack overflow - during stuck thread. Jarvis will reboot himself after running 5000 times on sentry_mode() changing the PID which resets the thread count.

Jarvis can:

• If anything apart from the below is requested, Jarvis uses google's Places API to match the requested phrase and suggest options. If the requested phrase doesn't resolve on Places API, it then uses google search engine parser to get results for the phrase. If anything beyond that, Jarvis will launch a google search on your default browser.

• Facial recognition and detection. Click here to read the setup instructions
• Wish you on events/festivals and birthday (requires an env var. example: birthday=01-January)
• Guard the surroundings using face detection and audio recognition. Sends an email and SMS alert in case of an intruder
• Increase or decrease master volume of your machine via voice commands
• Locate, ring and enable lost mode on any of the user's apple devices
• Control smart lights in the same IP range
• Read outlook calendar and inform about meetings in the next 12 hours
• Tell public and private IP address of the running device
• Swap voices on demand
• Adjust Screen Brightness
• Tell system vitals including fan speed, CPU and GPU temperature and boot time
• Restart the running device and suggest a restart in case of high boot time
• Mount and Unmount volumes connected
• Trigger personal_cloud with secure username and password (generated during runtime) sent as SMS
• Scan local ip range to get all the devices connected - acts as IP feeder for smart lights and TV
• Tell random facts
• Random heads or tails
• Control your TV
• Can scan and connect/disconnect to near by bluetooth devices
• Send a text message to most US based cell numbers
• Take notes and saves it to a notes.txt file
• Tell a joke
• Tell the number of repositories on your GitHub and clone a particular one
• Print the list of google home devices in your IP range
• Set up a reminder at a said time and sends a message to your phone at the said time
• Show the list of alarms on screen and deletes the one requested
• Set an alarm with the desired time, AM or PM is mandatory
• Get you the distance between two places or distance from your location to a particular place
• Create a database named tasks.db for TO-DO
• Add a new item and category as a task in your todo (Also tells if you it is a duplicate)
• Remove existing tasks from DB
• Get all the items in tasks table
• Delete the entire database
• Scan your Music folder for .mp3 and plays them (uses Windows Media Player by default for Windows OS)
• Look for unread emails in your gmail account and speaks the sender and subject line alone
• Initiate chatter bot (currently doesn't support Windows OS)
• Locate your iPhone and also provides options to ring the phone and enable lost mode
• Get your current location
• Get the weather information for your location (tracks location using your IP)
• Get the news update from fox news (news source can be changed in the news() function)
• Get your investment details using Robinhood api (Refer notes for setup)
• Get your system hard drive
• Get facts from Wikipedia using wiki api
• Open up the webpage you want in your default browser
• Open any application installed in running machine
• Get the current date and time
• Speak the current date, time, weather and latest news
• Open a google search for any query
• Repeat what ever it hears
• Shutdown the running device
• Respond to most basic conversations
• Put itself on sentry mode and wait to hear back from the user to wake up

Environment Variables:

• api_key - OpenWeatherMap API key
• news_api - News api key
• user, pass, qr - Robinhood Username (user), Password (pass) and QR (qr) code
• icloud_user, icloud_pass - iCloud credentials. Add recovery=<RECOVERY_PHONE_NUMBER> if you like to activate theft mode
• gmail_user, gmail_pass - Gmail credentials. Requires access to less secure apps or app specific access can also be enabled
• remind - Phone number to which reminders would be sent
• maps_api - For phrases that do not match existing features

Please note that these environment variables are required only if you need the mentioned features. If not Jarvis can work with less features.

Windows OS:

• To run Jarvis on a Windows OS, make sure pip3 command works in your cmd
• With current state of things, Jarvis lacks chatbot functionality due to version conflicts in requirements

Notes:

• If you want Jarvis to speak the weather info, you need an api key from OpenWeatherMap
• If you want Jarvis to speak today's news, you need an api key from News API
• If you want Jarvis to speak your investment details, you need to have a Robinhood account and follow the below steps for authentication purpose
  • Login to your Robinhood Web App
  • Go to Account -> Settings or click me
  • Turn on Two-Factor Authentication
  • Select “Authentication App”
  • Click “Can’t Scan It?”, and copy the 16-character QR code
• Jarvis uses a pre-trained model for punctuations. The model file comes from my S3 bucket, which is downloaded automatically when the file is not found

Facial Recognition:

• For this feature to work, a little bit of setup is required
  • Create a parent directory named train and create more sub directories with the name of a people within the parent directory train
  • Now drop the images directly within the sub directories named after each person
  • Please note that even if the sub directories are not added, the face_recognition script is written in a way that it can learn from unrecognized/new faces by storing them with named sub directories within train
  • The camera id has been made dynamic so that it can choose the camera automatically
• Try to keep the images as light as possible (which means both the display and file size of each image)
  • Reduced image size will not affect the accuracy as the facial recognition script converts each image into a pixel matrix
  • Reduced image size and display area can actually help in faster scanning and recognition
• Make good of the tolerance level in the face recognition script
  • The learning_rate can be switched to match your needs
  • Learning rate generally depends on the clarity of the images stored and the lighting at which it was taken
  • Try to avoid images with very low resolution to maintain an adequate learning rate
  • Lower the learning rate, higher the tolerance level (which means the lower you go on learning rate, the stricter or perhaps more accurate matching it does)
  • Leeman's terms:
    • Increased learning_rate: Exact to No match
    • Decreased learning_rate: Incorrect to close match
    • 0.5 - 0.7 should suffice in most cases though

The API keys can be generated for free once you sign up except for Robinhood, which you know how it works. If not probably you shouldn't be using it.