native iOS port of mobile client

[jywarren]

i was thinking I'd try wrapping the spectral workbench platform in PhoneGap and patch camera access in to replace the mediastream API used on Opera/Android. That may end up being harder than I expect...

Which is to say, the current mobile version is web-based, and only runs on Opera for Android right now. But it is pretty nice: http://spectralworkbench.org/capture

My ideal is to wrap this already-working system in a native app so that any future interface changes can simply be pushed out on all platforms at once.

[gitizenme]

Have you considered using Appcelerator Titanium? Check out: http://appcelerator.com

I have experience in coding Apps on both Titanium and PhoneGap and IMHO Titanium is the better of the two!

[jywarren]

I haven't -- would be interested to see the video frame access API. Is AT open source or compatible with? That's what's nice about PhoneGap/Cordova

[gitizenme]

The SDK and the IDE are open source - https://github.com/appcelerator

I'm not sure what your requirements are for video frame access are but writing a native iOS module to get at the video frames from the camera is one possibility: http://docs.appcelerator.com/titanium/2.1/#!/guide/iOS_Module_Development_Guide

[jywarren]

Darn, iOS 6 will support camera capture (so we might be able to do a single-shot version) but not getUserMedia video access: http://blog.merge.nl/shorts/web-app-html5-camera-capture-getusermedia-file-api-browser-support

[gitizenme]

I have not had time to poke around in the code. Does the analysis require a realtime video stream or can it be feed a series of captures?

[gitizenme]

Okay, I spent a few minutes looking at the code. This lead me to the underlying video capture library - WebRTC. I think I see a path from there to PhoneGap and/or Appcelerator.

[jywarren]

Cool! Sorry I've been scrambling and haven't been able to check in.
Basically we need access to raw image data, where we can get the R, G, or B
value of any pixel from the latest frame of video, in JavaScript. The rest
will be easy to integrate.
On Aug 30, 2012 6:07 PM, "gitizenme" notifications@github.com wrote:

Okay, I spent a few minutes looking at the code. This lead me to the
underlying video capture library - WebRTC. I think I see a path from there
to PhoneGap and/or Appcelerator.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/116#issuecomment-8176332.

[jywarren]

We could rejigger the interface for stills but live video is much much
nicer.
On Aug 30, 2012 6:07 PM, "gitizenme" notifications@github.com wrote:

Okay, I spent a few minutes looking at the code. This lead me to the
underlying video capture library - WebRTC. I think I see a path from there
to PhoneGap and/or Appcelerator.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/116#issuecomment-8176332.

[jywarren]

We are asking this question basically, and probably need to develop a PhoneGap plugin: http://stackoverflow.com/questions/10909748/getusermedia-shim-for-phonegap-cordova?rq=1

I'm looking into the barcode plugin since I imagine it does something similar -- intercepts video capture without showing a video interface widget, and runs some analysis on the video stream by pixel...

[jywarren]

Super -- more investigation shows that the Barcode plugin (which the more I think about it, the closer it seems architecturally to what we want to do -- record intensity variation in a horizontal row of pixels) uses the AVFoundation API in iOS:

https://developer.apple.com/library/mac/#documentation/AudioVideo/Conceptual/AVFoundationPG/Articles/04_MediaCapture.html#//apple_ref/doc/uid/TP40010188-CH5-SW2

which allows for "silent" video capture and processing. We gotta look at that barcode plugin closer.

[seandoyle]

I've just installed the latest phonegap on my Mac & I'll download the barcode plugin. I don't have any access to an Android device at the moment - but I was wondering if it would make sense to use OpenCV in the plugin if it provides a uniform video API for iOS and Android:
http://docs.opencv.org/trunk/doc/tutorials/ios/video_processing/video_processing.html
I have no particular reason for pushing OpenCV except that I've been looking for an excuse to learn more about it :-).

Is there any value in the additional image processing libraries that this would offer or is it just additional weight?

[jywarren]

I dunno much about OpenCV, but we were planning on writing a lot of the
image rec and matching code in JavaScript so it would be portable to other
devices. That said, if you can bake OpenCV into the codebase and it's a
more standard way to fetch video frames than writing our own interface
code, i don't see a problem. It wasn't clear to me if we could simply copy
the barcode codebase and just tweak it to get spectral intensity, but it
sounds worth a try.

On Sat, Oct 6, 2012 at 2:08 PM, Sean Doyle notifications@github.com wrote:

I've just installed the latest phonegap on my Mac & I'll download the
barcode plugin. I don't have any access to an Android device at the moment

• but I was wondering if it would make sense to use OpenCV in the plugin if
  it provides a uniform video API for iOS and Android:

http://docs.opencv.org/trunk/doc/tutorials/ios/video_processing/video_processing.html
I have no particular reason for pushing OpenCV except that I've been
looking for an excuse to learn more about it :-).

Is there any value in the additional image processing libraries that this
would offer or is it just additional weight?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/116#issuecomment-9200670.

[seandoyle]

I'll have to look at this again tomorrow - I'm having trouble getting the app running. The Cordova distro needed some massaging under Xcode 4.5 (remove the armv6 and armv7s architectures) and a bunch of other small things - and now it's launching but on both the simulator and iPhone 4 it's only partly starting. The last console message is Multi-tasking -> Device: YES, App: YES which is from the CDVViewController - but the javascript initializations don't finish.

My assumption is that I should uninstall cordova-lib and start again.

I've only taken a brief peek at the code underneath - but my instinct is that you're right that this would be a good template and that OpenCV might be overkill and add some complexity.

This should be simpler than it's been which probably means I'm doing something wrong :-).

[jywarren]

Moving this issue mostly to #174

[jywarren]

After much discussion we'll be focusing on our HTML5 web app; thanks!