Rendering Proteins and Molecules on AR
An ARKit iOS project for HackInOut 2017 (Devpost link)
This is a code bump of scripts, services and IOS Project for HackInOut 2017 submission. Please ignore the shammy code commits.
What does the App Do?
The App extracts chemical names from a large text in real-time and renders the molecules/proteins in an AR environment.
- Get real-time text from iOS camera using OCR module (AbbyyRtrSDK.framework).
- Find Chemical compounds/molecules/proteins from text from ML Modules.
- Wit.AI for processing keywords/traits and extracting entities
- Locally trained CoreML model instead of Wit.AI (TODO)
- Fetch the chemical name and search for information on ChemSpider
- The backend service takes a .MOL file containing the molecular data and renders into a 3D scene with OpenGL+PyMOL+MeshTool.
- The rendered .DAE (Collada) files are used to project Chemical formulae on AR Scene (using ARKit) in iOS App.
Demo link (YouTube)
ChemSteet iOS App includes
AVCaptureSessionis used for preview the iOS Camera layer for text recognition.
- Text recognition is done using ABBYY Real-Time Recognition SDK.
- On recognition of stable text input from camera & ABBYY SDK, locally it is processed for API request.
- API call for fetching list of chemical elements/molecules/proteins are obtained by Wit.AI backend which helps in mapping local DAE models for ARView.
- On selected chemical to view in AR, AR module is initialised for rendering local DAE model to display on ARView
ARWorldTrackingConfigurationfor rendering AR model (which in the form of DAE format) for the chemical chosen.
renderScene()render's with geometrical orientation & rotation of the AR model which is mapped for chemical (DAE chemical model present locally in the asset).
- Additional information of the chemical model such as
average_mass, molecular_weight, molecular_formula & common_nameare displayed along the AR model in Scene using
ChemSteet flask service includes
mol_to_deb_service for processing extracted text and serving molecule data from ChalkSteet API.
mol2dae.workflow an OSX AppleScript to automate .mol to .dae conversion. (
daes/ has all the extracted Daes files that can be rendered on AR.)
ChemStreet AI service includes
The entity recognition model etc will be uploaded soon.
[ ] Cleanup the project for contributions.
[x] Make a better readme page.
License and Copyrights
Copyright  [Sricharan Chiruvolu]
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.