Skip to content
detect and translate objects using CoreML & Vision
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
images
isla.xcodeproj
isla
islaTests
islaUITests
README.md

README.md

isla

Overview

Breaking language barriers with the tap of a button.

International travel is becoming more prevalent in an average person’s life. Language barriers should not hinder one’s ability to effectively communicate and to be fully immersed in another culture. isla aims to help people learn new languages more effectively and efficiently when traveling abroad. Using CoreML and using Yandex's Translate API, isla will be able to identify and translate any object in real time.

While isla is initially focused on the uses of an average traveler, it can also be expanded to the everyday language learner. Its ease of use and adaptability allows anyone to use the app and to cater it to their daily lives. With isla, we can make learning languages easier by allowing users to customize exactly what they want to learn.

Workflow Design

flowchart

Languages Available

languages

Buttons

buttons

Libraries

CoreML and Vision

  • Using Inceptionv3 model
    • From Apple’s Developer page: “Detects the dominant objects present in an image from a set of 1000 categories such as trees, animals, food, vehicles, people, and more.”
  • Vision ➔ framework that works with CoreML (classifications, object detection)
    • Display classification with the highest probability
  • How detection works
    • Capture current frame of the ARSCNView
    • Stores current pixels in a CVPixelBuffer
    • Converts buffer to CIImage
    • CIImage is fed to the Vision Request

Yandex API

  • Online machine translation service
  • Steps
    • Use API key and encode URL
    • Perform an HTTP request (GET)
    • Receive in JSON format, then parse it
  • Use of DispatchGroup to synchronize threads Asynchronous code

AVFoundation

  • Framework for time-based audio + visual media on iOS, macOS, watchOS, and tvOS
  • In this app, utilized to make Siri do text to speech
  • Steps
    • AVSpeechSynthesizerVoice() ➔ choose Siri’s voice
    • AVSpeechUtterance() ➔ Siri will speak
    • AVSpeechSynthesizer ➔ prepare for sound
    • Speak!

Screenshots

home main dictionary

You can’t perform that action at this time.