Skip to content
/ arva Public

Augmented Reality Virtual Assistant for UNBC

Notifications You must be signed in to change notification settings

haskins/arva

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

2 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

ARVA

Augmented Reality Virtual Assistant for UNBC

User Manual

###Preface

Welcome to ARVA!

ARVA stands for Augments Reality Virtual Assistant. This application is designed to be used at UNBC to aid in locating information about rooms and office hours, and also to give you, the user, a guide of UNBC.

This application was designed as part of an Undergraduate Research project in Computer Science.

We hope that you enjoy using our application.

deJovi

aka Devon Harker, Josh Haskins, & Vincent Tennant

###Navigation Navigation on this application is accomplished by using the touchscreen and by using the hardware button on the phone being run on. Listed below are the different menus that are available and the functions available inside of this application. ####Main Menu From the main menu you are able to go to the Help menu, or proceed to using ARVA. ####Help Menu This menu allows the user to learn how to properly use ARVA. Use the back button on your device to go back to the Main menu. ####Room Information This application also has the ability to scan in targets and reveal the room information, such as office hours for a professor. ####Scavenger Hunt Throughout UNBC there are targets that have been set up to be used for the Scavenger Hunt. When a target has been recognized, a chime will be heard though the device’s speakers and a 3D coloured star will appear on the screen. Moments later the data related to the target will be displayed onscreen. ###Troubleshooting ####Launching the Application If the application does not launch, try restarting your phone. If that does not fix it, try uninstalling and then reinstalling the application. Also ensure that you are running a device that meets the system requirements for our application.

####Displaying Information If the application will not display target information, ensure that you have a connection to the Internet. Then try quitting the application and re-launching it. ###FAQ ####What devices does the application work on? The application should run on any Android phone or tablet. It is guaranteed to work on a Nexus 4.

####Do I need an Internet connection? Yes, Internet is required to retrieve any information about the targets scanned.

####Do I need a camera? Yes.

####What version of Android do I need? Android 2.2 or higher is required.

##Technical Report ###Problem Description Augmented Reality, or AR for short, is a rapidly progressing technology that allows users to superimpose virtual objects into the real world. We can use AR as a medium to convey helpful information about reality and this information can aid people in many different situations, not simply just at UNBC. Finding information about UNBC can be difficult, as many different websites have to be used and sometimes have conflicting information. The solution is to have an app to be a repository for all information and pull in additional data from the UNBC website. The application is targeted towards new students and visitors to the UNBC campus. There is a self-guided tour element of the application, which encourages the user to explore the UNBC campus. Inside of this tour, there is an aspect of collection, similar to a scavenger hunt; when all scavenger hunt targets have been found, the hunt is complete. This hunt encourages users to explore the UNBC campus, and go to places they may not regularly visit. ###Solution Approach While it is not a new topic, little has been done with AR at UNBC. With the consumer-held virtual reality coming in the near future, AR is also becoming a hot topic. As such, we decided it would be a worthwhile endeavour to do research into this topic. We hope to lay the groundwork for AR at UNBC and inspire future students to expand on what we have started. There are currently some AR applications available on mobile platforms. Though the goal and design for these applications are specific to niche markets, such as home furniture visualization and novelty applications. While useful for very specific tasks these applications don't seem to take full advantage of AR technology. Our goal is to lay the framework for an AR application targeted towards UNBC, and to allow for the application of our work to other universities and businesses. The goal of this project was to explore AR, the practical applications of it, and how students and visitors who attend UNBC can use it. Using Unity 3D to develop our project we gained the ability to very quickly and easily release our application to multiple platforms. Developing our application for multiple platforms would allow for greater exposure and use of our application. By using AR, we are able to convey a near infinite amount of information to the user. Unlike QR codes, which can only hold 7089 characters, our application consults an online database to retrieve information about the target [1]. ###Solution Design The main focus for this application is to provide an immersive and interactive experience for newcomers to UNBC. Using AR to change the way that new students and visitors see the university can drastically affect the image of UNBC in a positive way. This application will show that UNBC is more than just a green university [2]; it is also a leader in technical innovation. One of the clearest goals we had for this project was to not simply make it a glorified QR code reader type application. We wanted to take full advantage of the tools that AR could provide. Overlaying 3D objects in real time, while providing relevant information to the user. The information provided will be relevant and useful to everyone at UNBC: class times and room schedules will be displayed, room bookings for library rooms, professor information is accessible outside of their offices, menus for food establishments, and event information will all be accessible to the user. Making this information as clear and concise as possible is imperative. Not everyone has an 11-inch tablet, or 29-inch 4k monitor. It is necessary to keep the important information, making it clear, and ensuring that it fits nicely and cleanly on smaller screens to allow everyone with a smartphone or tablet can access this application. ###Tools Used For our project three main tools were used: this includes Unity, Vuforia and Parse Core. Both Vuforia and Parse were used as plugins inside of Unity to use their features in our application. ####Unity Unity is a powerful cross-platform game engine that can be used to create games and many other applications. It has a near infinite number of tools that can be utilized to create your application or game. The editor is very customizable, allowing many different types of views depending on what your current task is. It is similar to Eclipse, where it has many different parts to it, trying to be an all in one IDE, which it does very well. The only external application that Unity uses is MonoDevelop to edit the scripts. Initially being announced as only for Mac in 2005, it has expanded in recent years. Unity supports development of applications and games on the following platforms: Windows, OS X, Linux, Xbox 360, Xbox One, Wii U, PlayStation 3, PlayStation 4, PlayStation Vita, Windows Phone, iOS, Android, BlackBerry 10, Tizen, Unity Web Player, Windows Store, WebGL, Oculus Rift, Gear VR, Android TV, and Samsung Smart TV [3]. Just recently it has been announced that Unity development will be supported for the New Nintendo 3DS and New Nintendo 3DS XL [4]; which further expands the range of platforms available for developing and publishing on, using just one piece of software. There are two different versions of Unity, Personal and Professional. The Professional features are as follows: engine with all features, royalty-free, all platforms (with some limitations), customizable splash screen, Unity Cloud Build Pro - 12 months, Unity Analytics Pro, team license, prioritized bug handling, game performance reporting, beta access, unlimited revenue and funding, future platforms included, Professional editor skin, Asset Store level 11, and Professional community features [5]. For our project we used Personal, as we did not require any of the Professional features. Unity is actually very simple to get started with. To start of, simply create a scene. “Scenes contain the objects of your game. They can be used to create a main menu, individual levels, and anything else. Think of each unique Scene file as a unique level. In each Scene, you will place your environments, obstacles, and decorations, essentially designing and building your game in pieces.” [6]. Different types of components can be added to the objects in the scene. These components can be scripts, audio and much more. “Scripting is an essential ingredient in all games. Even the simplest game will need scripts to respond to input from the player and arrange for events in the gameplay to happen when they should. Beyond that, scripts can be used to create graphical effects, control the physical behaviour of objects or even implement a custom AI system for characters in the game.” [7]. We choose Unity based upon its ease of use and ability to port the software to a multitude of platforms. Also the Asset Store is location where plugins, 3D objects, audio and much more are available for download to use in the project, some for free and some being paid. We used this to our advantage to for 3D objects and for audio effects. ####Vuforia Vuforia is an Augmented Reality SDK that uses computer vision technology to identify and track image targets in real-time. Target ratings are of the utmost importance in target recognition. An augmentable rating defines how well an image can be detected and tracked using the Vuforia SDK. The augmentable rating can range from zero to five stars for any given image [8]. The higher the rating of the image target means the stronger the detection and ability to track. A rating of zero signifies extreme difficulty in tracking the target, whereas five stars is easily tracked.

There are a few different traits that make up a target. The first being features. A feature is a point on the target where the AR system is able to pinpoint. A square has four corners, meaning that there are four distinct features that can be tracked [8]. Whereas a circle has no edges, meaning no features to track. See Figure 1 showing where the features of these shapes are. Essentially, the more features a target has, the better the target is for tracking. Also the more balanced the distribution of features present in an image, the better the image can be tracked. Otherwise this can result in the augmented object jumping around on the screen when either the camera or the object is moved. Second is contrast; the contrast of the image can greatly affect the ability of the AR software to detect and track the target. Also clearly defined shapes and sharper edges aid in detecting and tracking. Compressed images, blurred images, and organic shapes do not provide good targets, as there is not enough richness in the target for the AR system to detect or track correctly. “Typically, organic shapes with soft or round details containing blurred or highly compressed aspects do not provide enough detail to be detected and tracked properly or not at all. They suffer from low feature count.” Third is patterns; repetitive patterns are unable to be tracked by AR systems [8]. A checkerboard is a very good example of a repetitive pattern, as it is simply alternating black and white boxes. The squares look exactly the same to the software, thus they are not able to be distinguished by the detector software, even though they are squares and have features.

One of the targets we tried to use for our scavenger hunt part was a view of looking up into the Echo Chamber, see Figure 2. The problem with this view is that it was simply just circles, which provide no features, so the AR system was unable to track and thus we abandoned this target. So in summary, what makes a good target? Plenty of features spread out throughout the entire image. In addition to having good contrast and no repetitive patterns. ####Parse Core Parse Core was used for our remote database. We chose Parse Core as it offered a free portion of its services and integrated easily with Unity. Parse describes Core as “[Core] handles everything you need to store data securely and efficiently in the cloud. Store basic data types, including locations and photos, and query across them without spinning up a single server.” [9]. To access the server data we simply used queries to search the database and return the information required. ###Detailed Implementation Our goal for this project was to create a mobile application for UNBC and make AR technology a major part of this application. When first starting out, Unity 3D appeared to be the clear solution to our problem. It allowed for very easy multiplatform development while at the same time making integration with AR SDKs simple. Instead of developing our own tools to be used for this project we choose to work with Vuforia. Vuforia is an AR SDK from Qualcomm. Major features of this SDK that appealed to us as it included a very efficient, and smooth tracking algorithm. Object-Character-Recognition allows Vuforia to recognise individual character or words, and even sentences. As well the Vuforia SDK allowed for very easy integration with Unity. To store all of the information that the application would require, we chose to use Parse, a cloud-based database system. Parse allowed us to simply and easily create an online cloud-based database to store relevant information for our application. Information such as schedules for professors, university services, food, and events are all stored on our Parse database. By using a cloud-based service to store and keep track of all of our information allows us to very conveniently keep all information up to date in our application without having to consistently release updates for new information. The database simply needs to be queried and the relevant information is returned. In the event that we, or others, wish to expand the application parse offers easy options for allowing for greater traffic loads. This makes it the ideal solution for storing out information. We chose to implement scripts in Unity with C# instead of JavaScript or Boo. These C# scripts are used for a variety of purposes, including functionality for accessing the Parse database, functionality for managing target acquisition and loss events, and for altering the GUI to display information. As the project progressed, we opted to put an extra bit of polish into the project, this included sound effects [10], [11], 3D objects [12], and a proper GUI. ###Communication Communication between different modules in our project is of great importance. To start off the Unity application is the main of the project, it handles all the user interactions with the application, and calls upon further modules to perform their actions. Inside of the Unity application we have implemented auditory feedback, touchscreen navigation and also button navigation. The Unity application then calls upon the Vuforia AR module to run. When the AR module detects a target, it returns this target’s identification number back to the Unity module. From there, Unity then calls upon Parse. Parse will query the database and return the result back to Unity. When Unity receives the information, it will update the on screen text of the graphical user interface to display the information. When the target is removed from the camera’s visibility, the AR program will clear the text on screen and display the ‘waiting for target’ message. ###Experiments and Results ####Database Parse offers a number of different data types that can be used for relational databases such as pointers [13]. Since the data types differ slightly, we had to experiment with them to find which one suited our project the most. We determined that it was simpler for us to not use any of the relational types; instead we opted to use only primitive data types. ####Character Recognition Vuforia offers Optical Character Recognition (OCR) and we experimented with it to determine if it could be useful for our project. We were able to get OCR to work on a variety of words such as names and found that it could distinguish between words where the only difference was that one of the words began with a capital letter. It was also able to perform OCR on hand printed letters. Unfortunately, Vuforia was not able to perform OCR on numbers [14], which we thought was extremely strange and unfortunate. We did research into other software or Unity plugins that would allow us to implement this aspect into our project, but we were unable to find one that met our criteria. ####Target Quality Not all targets are equal in terms of how recognizable they are to a camera. We discovered that Vuforia is able to rate a target based on how recognizable it is and it assigns a rating between 0 and 5 stars inclusive [8]. Using this, we were able to experiment with various targets and we were able to determine some of the qualities that cause a target to be highly rated. ####Unity Unity is a very powerful engine and it is capable of many different forms of development types, such as games and applications [15]. However, having never used it before, we spent a large amount of time exploring Unity and determining which aspects of Unity would be useful to our project.

###Challenges Faced Throughout this project many different problems were faced, some that halted the project for a brief period of time, while others we worked around until a solution was found. The major problems encountered are as follows: ###Qualities of Targets Ensuring that Vuforia was actually able to detect the desired targets proved tricky. Initially we planned on using room number plates outside of each room to allow for the detection and retrieval of relevant information for a specific room. At first it seemed to work well but we quickly realised that the room number plates themselves were too similar to be easily distinguished between by Vuforia, see Figure 3 for a sample plate. Vuforia would detect the room number and display the information for another room, making the use of these room numbers not possible. Target symmetry was also an important feature. Making sure the desired target could be detected at any angle, while at the same time providing distinct lines, and high contrast; all to ensure that the target is very easily detected by Vuforia. In reality it is not as easy as it seems to find static objects to use as targets that check all of those boxes. As well as making sure that the targets we choose are easily detectable by Vuforia, we had to make sure that the targets we choose to begin with actually stay where they are and aren’t removed anytime in the near future in important. Making the main target for the Thirsty Moose a poster that was hung up a week ago is not an ideal target. Though the poster itself may be a very ideal target for Vuforia, the poster may be pulled within a few days making the target useless. For temporary events, posters may make great targets, but for more permanent solutions static targets are required. ####OCR on Numbers A possible solution to the problem of Vuforia not being able to differentiate between different room number plates was to use Vuforia’s built Object Character Recognition software to detect the actual room number. This would enable us to very easily uniquely identify each room number. The problems arose when we learned that Vuforia’s OCR software only recognizes capital and lowercase letters, and does not accommodate number detection [14]. We did conduct a large amount of research on this topic, but where unable to find any solutions that would be able to be used with our project as it. What constantly kept on coming up in regards to Vuforia was “Vuforia does not support number recognition at present” [16], said multiple times by AlessandroB whom is a moderator on the Qualcomm Vuforia Developer Forums. As for OCR on numbers in our application, we opted to continue with the project idea, but discard the room recognition portion of the application using OCR. ####Script IDE To edit C# scripts in Unity, we used MonoDevelop [17]. Unfortunately, the MonoDevelop application that was bundled with Unity was out-dated and featured bugs that were frustrating to work around. One of the more prevalent bugs that we were forced to deal with was that the compiler would occasionally find issues with statements that we were certain were syntactically correct. We discovered a simple workaround: by copying and pasting the code that the compiler thought was incorrect over top of itself, the errors simply disappeared. This was simply just an incompatibility between MonoDevelop, Unity and the computer operating system. An updated version of MonoDevelop has been released, but unfortunately Unity uses a slightly customized version of MonoDevelop to ensure that scripts are synced back to Unity once editing is complete [17]. ####Updated Software When we started development on the project, Unity had just released version 5 of their IDE, we were excited to try it. Unfortunately this took a bit of troubleshooting to end up figuring out that Vuforia 4.0.103 does not work 100% with Unity 5. So we reverted back to Unity 4, and did not encounter any more of these issues. Though, a few weeks later Qualcomm released 4.0.105 to solve compatibility issues with Unity 5 [18, p. 5]. We opted to continue using version 4 of the software as we had already began our project, and did not want to have to learn a slightly new interface. Another software issue we encountered was that the main computer we were using for development automatically updated the OS version to Mac OS X 10.10.3, which caused a problem with our project, mainly just the usage of the webcam view through Vuforia. After a lot of testing it was determined that simply creating a new Unity project and importing all the assets from the ‘broken’ project would solve our problems. We did this and were able to continue working on the project. ###Evolution With additional time, we would like to implement position tracking with the use of GPS. This would be very useful as it could allow the app to act as a map for those who are looking for a specific location. We would also like to migrate the app to Android Studio. This would allow us to have increased flexibility through the use of additional tools and we would no longer be restricted Unity. Tools such as GPS, and location services. Android Studio would also allow us to design a better UI, inline with Google’s Material Design guidelines [19]. ###Learning Outcomes Over the course of the project, we learned how to use some of Unity's features including; how to use the Unity Asset Store, how to script in C# using the MonoDevelop IDE, and how to use plug-ins such as those that provided Vuforia and Parse. Despite having never used C#, we were able to write a number of scripts and to modify existing scripts to better suit our project. We also learned the basics of AR and how to design software that uses AR with the use of Vuforia. We feel confident that we could take what we have learned and apply it to other AR engines. In previous courses we have learned the basics of databases and how to work with them. In this project we were allowed the opportunity to gain additional practical experience with databases. We learned how to use Parse to manage a remote database. We learned the specifics of Parse and how best to implement the database that our project needed using its feature set. We learned that it is important to be flexible; for example, we revamped our project to accommodate for the fact that that number recognition was infeasible. ###Compiling the Program To compile the project, simply open the supplied Unity project inside of Unity 4.6.4. The required plugins for Parse and Vuforia are include inside of the project package. This project was tested running natively on a Nexus 4 running Android 4.4 and was streamed to an iPhone 6 running iOS 6.3 using Unity Remote. An already complied Android APK has been supplied, ready to be installed on any Android device.

About

Augmented Reality Virtual Assistant for UNBC

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published