feature : official 64bit support in osx

[elliotwoods]

Success condition is either:

• "My mac 64-bit" scheme in XCode
• "osx 64bit" as an option in Project Generator

[roymacdonald]

I can't remember if the 64 bit support for OSX was ready. Is it?
All I remember is that the quicktime section was the one holding back and that @obviousjim was implementing it.
Having the 64 bit build in PG would be great.

best!

[elliotwoods]

I presume the issue is video (as always.
I'm a tentative +1 for 64bit becoming the default osx option within the next year.
Getting all addons to switch will be a bit of a pain, and there hopefully will be some clean ways to implement this (e.g. universal .dylib's or funky project macros in the build process)

[roymacdonald]

I'm a bit ignorant about switching from 32 to 64 bits. Is the issue having precompiled libraries within an addon? If an addon is just uncompiled code (.h and .cpp files) there shouldn't be any trouble compiling it for 64 bits ?

[elliotwoods]

that's correct. you need the correct architecture for any code which you are linking into your program for which you're not building from source in your project (e.g. Poco, GLUT, OpenCV, libfreenect, libdc1394, etc)

in many of these cases their is either:

1. equal availability of pre-built 32bit and 64bit libs
2. source code available so we can build 64bit ourselves

but this isn't always the case (e.g. commercial libs)
sometimes there's only 32bit or 64bit versions available

a .dylib is a bunch of linkable compiled code, and can contain both 32bit and 64bit versions within the same file, so ideal is for core and all addons to supply universal .dylib's for each library instead of 32bit versions (eventually dropping 32bit).

[roymacdonald]

@elliotwoods thanks for the clarification.
Just like I presumed.
What are you thinking when you say "funky project macros in the build process"?

[obviousjim]

One way to move forward on this is to implement the AVFoundation video
player. The OpenGL path for this feature requires 10.8+, so that'll make a
sticky double release for OS X for while

Alternatively we can look at GStreamer, which may be the way to go as
it provides better codec support and would also fix our windows QT
dependency. The processing folks go this work in a distributable way - the
challenge is not introducing any more install dependencies or bloating the
download significantly... but i'm leaning this way for the time being...

Working on the Clouds documentary one of our big challenges is HD playback
on windows, so we'll either go with Gstreamer or need to implement a
Windows Media SDK based player (which should be 64 bit as well)

On Fri, Apr 19, 2013 at 10:38 AM, Roy Macdonald notifications@github.comwrote:

@elliotwoods https://github.com/elliotwoods thanks for the
clarification.
Just like I presumed.
What are you thinking when you say "funky project macros in the build
process"?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2016#issuecomment-16656442
.

• James

[vanderlin]

With 0.8 is this now possible to have a 32bit and 64bit build setting in one xocde project?

[c-mendoza]

I'm getting to this a bit late, but is there any thought being given to potential video capture problems in an OS X 64 bit release? As you may know, traditional VDIGs were deprecated in the move to QuickTime X, which meant that they are only accessible if an app is 32 bit mode, and most (if not all) non-Apple capture devices are unusable or not visible in 64 bits. The problem is even worse in AVFoundation, at least in my testing, since no legacy capture devices are visible even if the app is 32 bits. This includes any VideoGlide devices and all of the Blackmagic capture devices I've tested. DV transport-type devices (like the ADVC 55) do work because they rely on DV, which comes with the OS, but they have a relatively high overhead and are limited to one per system.

[obviousjim]

Working with @prisonerjohn and @kronick we've created a near feature-complete AVFoundation replacement for the QTKit based player: https://github.com/obviousjim/ofxAVFVideoPlayer

The video player is the final piece holding OS X back from being 64 bit compatible. In hopes of getting the AVF player into the core, I'd like to start discussing how we could structure the 64 bit release a bit differently on OS X.

As @arturoc and I discussed at YCAM there are some fundamental things that we could change with the next iteration of the video player API. Specifically relaxing the guarantees on synchronous behavior with frame access, and possibly creating a different object for synchronous video processing operations versus playback.

I'd like us to consider separated 64bit/32bit releases on OS X so that the 64 bit release could serve as an test for this approach, while maintaining the backwards compatibility on 32 bit builds. If we could remove the requirement for synchronous behavior on video players (which is what made the QTKit integration so complicated) I think we could get a release out pretty quickly.

cc @bakercp should definitely weigh in.

[ofTheo]

@NickHardeman has all the libs built for universal 64bit/32bit OS X

It seems relatively straightforward with Apothecary.
Might be a good opportunity to take another pass and build with libc++?

So at least 0.8.1 can be libc++ friendly.
We could then do an experimental release which defaults to 64bit on OS X.

Or even better ship one release and include different targets.
One of which builds 64bit and based on ENV VARS includes the right files / headers in the project.

edit: changed C++11 to libc++ which is what I meant to say.

[ofTheo]

pps: I think this is totally doable, the main thing we were waiting on was the video player.
the libs and xcode targets should be fairly easy.

[obviousjim]

Thanks for the quick reply @ofTheo!

@prisonerjohn and I playing with @danomatika's script right now to get a build. If we get it we'll post a branch of OF that can serve as a starting place for this. @arturoc and I should sync up and get a basic version of what we prototyped at YCAM implemented with the new AVF player swapped in.

Also I think it would be wise to try to take the AVF Asset Reader approach that the iOS player uses and try to convert that into a class for doing synchronous video processing, and maybe replace the current iOS player with a better implementation based on the desktop player.

All a bit down the road, but having a 64bit branch to experiment with will be exciting!

[ofTheo]

I think we can move even quicker. I would like to get at least 32bit/64bit universal libs in the next release.

The filenames wouldn't change so it would be an easy switch.

Sent from my iPhone

On Feb 2, 2014, at 3:49 PM, James George notifications@github.com wrote:

Thanks for the quick reply @ofTheo!

@prisonerjohn and I playing with @danomatika's script right now to get a build. If we get it we'll post a branch of OF that can serve as a starting place for this. @arturoc and I should sync up and get a basic version of what we prototyped at YCAM implemented with the new AVF player swapped in.

Also I think it would be wise to try to take the AVF Asset Reader approach that the iOS player uses and try to convert that into a class for doing synchronous video processing, and maybe replace the current iOS player with a better implementation based on the desktop player.

All a bit down the road, but having a 64bit branch to experiment with will be exciting!

—
Reply to this email directly or view it on GitHub.

[obviousjim]

Sounds good, and if we can do the different targets you mentioned it could spare the double release package option, we'd just need to be clear about how it affects the functionality.

Also just found this from @NickHardeman:
https://github.com/NickHardeman/openframeworks_osx_64

Thanks for the tip!

[NickHardeman]

Aha, I was just going to share that link. :)
The only thing that needs some work is the video player. I started implementing an AVFoundationVideoPlayer, but then found this one: https://github.com/kronick/ofxAVFVideoPlayer so I never finished my implementation. I don't think it's in the repo, since I never finished it. I can post it if you think it would be useful, but you could probably just use the Kronick version

[obviousjim]

Hey @NickHardeman we are right with you. The https://github.com/obviousjim/ofxAVFVideoPlayer that we are using is a fork of @kronick's which supports 10.9 and pixel access. @prisonerjohn and I just forked your osx_64 branch and are adding our video player to it.

Working great so far!

[NickHardeman]

I just pushed up the video player into the addons folder. The apps/FATDaddy/64bit example is a project that is all setup.

[bakercp]

Just to throw in -- let's please use a 1.5+ Poco if we're rebuilding libs. I've got a bunch of unicode support ready to go into the core that we get w/Poco 1.5+. Poco 1.5+ means we don't need ICU or any other third party library to have the first class Unicode/utf8 support we planned in the roadmap. Also, there are a lot of network fixes, JSON, etc. I realize 1.5 is a development release ... but everything before 1.0 in oF land is a development release right :)

[obviousjim]

@NickHardeman + @ofTheo is it a problem that nick's of64 is not a fork of OF? Should we be working in that tree so we can do a pull request?

@bakercp we just forked https://github.com/prisonerjohn/openframeworks_osx_64 if you'd like to take a look at pushing the 1.5 libs into there that'd be awesome.

[NickHardeman]

The 64 bit version I was working on is from a download of OF 008. If it's easier, I can pull down OF and merge it into there.

[obviousjim]

Hey @NickHardeman , would you be down to take our video player instead? We can send you a pull request from @prisonerjohn's fork that we just pushed.

It'd be good to tag the 008 release and then find a way to get it back into the openFrameworks tree so that we can eventually open a pull request

[NickHardeman]

Yeah. I never finished my implementation. So send it on over. I can make a 008 tag so it's easy to find.

[obviousjim]

Cool we'll send you a pull request in a second

[obviousjim]

We're trying to get 32/64 building together in the same project -- we noticed that all the .a are split up. the Apothecary script that we were running creates universal libs with 32/64 in the same .a file, wondering if that would be easier.

In the meantime so we can work with this we are going make separate targets and modify the CoreOF.xcconfig to switch what libs it's pulling in depending on if 32 or 64 bit is being built.

@ofTheo @bakercp may have opinions on how we should structure this.

[ofTheo]

Would be good to get this into a branch of the main OF repo or a fork of the repo for sure.

We can look at doing 1.5 poco later. First it would be good to get the current libs working well as 64/32

If we make osx 1.5 then other platforms will be out of sync.

Main thing I think is to get this into a fork / PR so we can start to get things working.

Sent from my iPhone

On Feb 2, 2014, at 4:25 PM, James George notifications@github.com wrote:

Cool we'll send you a pull request in a second

—
Reply to this email directly or view it on GitHub.

[NickHardeman]

Ok. That's a good point. I already started building the 1.5.1 Poco libs. Right now if you build for 64 bit and switch to 32 bit, the OF project build arch needs to be switched. What is the best way to build the project and have it switch the build arch of the OF project?

[obviousjim]

I think we could make two targets for both oF and the app, and configure
the dependencies differently for each. That will let us also add defines
that can swap in the new avf stuff based on which target is being built.

First order is to get this into the main of tree so we don't lose all these
commits.

On Sunday, February 2, 2014, Nick Hardeman notifications@github.com wrote:

Ok. That's a good point. I already started building the 1.5.1 Poco libs.
Right now if you build for 64 bit and switch to 32 bit, the OF project
build arch needs to be switched. What is the best way to build the project
and have it switch the build arch of the OF project?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2016#issuecomment-33914262
.

• James

[NickHardeman]

@obviousjim I believe that the libs are FAT. I made a folder to store the older libs, so that I could revert easier.

[obviousjim]

Ok, can you confirm @prosonerjohn? Apparently libtool reported them as
being 64bit only. Will check myself in a second.

On Sunday, February 2, 2014, Nick Hardeman notifications@github.com wrote:

@obviousjim https://github.com/obviousjim I believe that the libs are
FAT. I made a folder to store the older libs, so that I could revert
easier.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/2016#issuecomment-33915170
.

• James

[prisonerjohn]

Looks like some are universal but most at the root of lib/osx are 64bit only.

Split:

freeimage
assimp
cairo
fmodex
freetype
glew
glut
rtAudio
tess2

Universal:

glfw
openssl
poco

[danoli3]

I have a Travis-CI Apothecary script building Repo that lists the OS X side as well as iOS formulas in their build status: https://github.com/danoli3/apothecary-den

Build Status:

Lib	osx	ios
FreeImage
FreeType
tess2
poco
openssl
ofxAssimpModelLoader -> assimp
ofxOpenCV -> opencv
glew		N/A
glfw		N/A
glu	N/A	N/A
glut	✗	N/A
cairo		N/A
kiss	N/A	N/A
portaudio		N/A
fmodex	✓?	N/A
rtAudio		N/A

---

This doesn't show complete coverage though, Table is built from a full coverage type found in the Apothecary directory.

[kylemcdonald]

@danoli3 thanks so much, that is super helpful for giving me an idea about where we are. "passing" means that both 32 bit and 64 bit are compiling, or is it just that the formula doesn't have any errors for whatever it's doing at the moment?

[danoli3]

So the passing means it is successfully compiling the static libraries for i386 libstdc++ and x86_64 libc++ (64bit) for OS X. (And i386, x86_64, armv7 and arm64 for iOS).

This gives a good indication of where the libraries are at, however not the complete story.
For example; With iOS it had heaps of Symbolic link errors at linking stage with DEBUG mode due to some strange scope issues the libraries had (i.e. using the DEBUG preprocessor internally pocoproject/poco#544 ). I've fixed these at the core of those libraries now, however this the the testing that will be required in the next stage for 0.9.

[kylemcdonald]

@danoli3 awesome. i also see now in #3453 (comment) that @ofTheo already encouraged the idea of merging formulas before they have complete coverage. good to see.

[danoli3]

@kylemcdonald Yeah master has all of the latest Apothecary Formulas we currently have.
Strategy was Strip any pull requests for Apothecary with static libs/updated headers, to just the core scripts themselves using cherry-pick.

So from now on, any new Apothecary formula updates / PR's:
Make sure to just have the pure apothecary script changes only.

PR's with static libs + updated headers can come in later in separate PR's.
(This decoupling has made building it all much easier as their was many inter script dependancies, especially for all the android scripts, and Poco + OpenSSL for all).

[obviousjim]

@kylemcdonald currently @prisonerjohn's fork of OF works on Windows and OS X, We are using it for projects right now on both OS's. What issues did you find in compiling it?

I lost track of understanding the complexities of merging this when it became dependent on Apothecary and c++ 11. @elliotwoods and @bakercp are more up to date on this aspect. But my vote is for getting whatever issues kyle is finding finding merged into the OF 64 branch, fixed up and publish how to get it running somewhere.

[tgfrerer]

hey @danoli3 i've tested glew on os x and it seems to build fine for me, is it possible that the travis build stage fails at clean() because the travis system doesn't copy the files upon copy ?

[danoli3]

@tgfrerer Yeah the Copy part is failing due to:
`rm: libs/glew/include: No such file or directory``

Will just need to add rm -rf (remove if exists) to fix this.

[tgfrerer]

I found out something quite exciting: clang can make fat libs with the correct -stdlib and -stdc++ in one go, while compiling!

Since this might speed up & simplify generating apothecary scripts considerably, i've put my findings into a gist. It contains the words "vanilla" and "fat", but not "ice cream":

https://gist.github.com/tgfrerer/8e2d973ed0cfdd514de6

I'll do some more testing on this tomorrow, but the initial fat libs i got were quite promising. I'm putting this up now so that folks in other time zones with more clang fu and linking experience might want to check up on it whilst i catch up on sleep.

[kylemcdonald]

going to un-assign @obviousjim for on this issue now. still ❤️ you, just want to open up the spot in case someone else wants to take the lead on this :)

@tgfrerer that's amazing, i would love to hear @danoli3 and @bakercp on this.

also, regarding my effort to get 64-bit compiling, it ended up working with @prisonerjohn's branch. i just had to craft a project file that was correct. using a pretty standard project file i saw this from ofApp:

ld: warning: ignoring file /Users/kyle/Documents/openFrameworks/libs/openFrameworksCompiled/lib/osx/openFrameworks.a, file was built for archive which is not the architecture being linked (i386): /Users/kyle/Documents/openFrameworks/libs/openFrameworksCompiled/lib/osx/openFrameworks.a
Undefined symbols for architecture i386:
  "ofSoundBuffer::operator[](unsigned long)", referenced from:
      ofBaseSoundInput::audioIn(ofSoundBuffer&) in main.o
      ofBaseSoundOutput::audioOut(ofSoundBuffer&) in main.o
  "ofSetupOpenGL(int, int, ofWindowMode)", referenced from:
      _main in main.o
  "ofRunApp(ofBaseApp*)", referenced from:
      _main in main.o
ld: symbol(s) not found for architecture i386
clang: error: linker command failed with exit code 1 (use -v to see invocation)

i was a bit confused about why it mentioned ofSound objects... but this made me realize the ofApp was still building as i386 somehow.i looked in the build settings it said "Native architecture (x86_64)" but i switched it to "64-bit Intel (x86_64)" just for fun. it worked! then i switched it back to "Native" and it worked this time.

so it was a one-toggle switch but i'm not clear why it wasn't working with the original option, some kind of object file cache that needed to be cleaned and rebuilt?

[pizthewiz]

I found out something quite exciting: clang can make fat libs with the correct -stdlib and -stdc++ in one go, while compiling!

Definitely a great find and could simplify the build process, but I wonder if some of the dependencies use their own build systems such that we aren't providing the actual args and arg ordering?

[bakercp]

@pizthewiz Yes, definitely an awesome find, but indeed many of the packages are using their own make system settings, etc and it's sometimes difficult to modify a build system (e.g. Poco e.g. pocoproject/poco#680).

[danomatika]

Also, how will this work with the iOS simulator? Does the iPad 2 simulator use 32 bit (matching iPad arch) or 64 bit libs (matching OS X)? If so, it may not be as easy as saying 32 bit iOS libs are C++ and 64 bit are C++11. I don't know but I thought I'd throw that out there.

[pizthewiz]

Also, how will this work with the iOS simulator?

The iOS simulator uses the libs paired to the target device arch - simulating the iPhone 5, a 32-bit armv7s device, will use 32-bit i386 libs and simulating an iPad Air, a 64-bit arm64 device, will use 64-bit x86_64 libs.

[danomatika]

... so that means we need i386 libs with both C++ & C++11 std libs? From what it seems like, if you want to use OF 0.9.0 with C++11 on an iPhone 5, you'll be linking to the i386 libs.

That makes me want to advocate for going completely cold turkey on iOS & OS X with the old C++ std lib. Or least make a huge deprecation warning until the next major release to justify the potential complications etc. Do these ultra-fat libs actually work?

[pizthewiz]

if you want to use OF 0.9.0 with C++11 on an iPhone 5, you'll be linking to the i386 libs.

Just to be a little more clear, if you wanted to simulate an iPhone 5 in the iOS Simulator, you'd be using the i386 slice of libs - on device, it would use the armv7 slice. The i386 libs used for the iOS Simulator and those for OS X are separate though - does iOS need anything but libc++ and C++11?

/cc @danoli3

[danomatika]

They are separate, yes (i.e. iOS fat libs include both arm and Mac arch). I'm just asking to make sure we don't miss this point and lead to link errors etc down the road. Best to double check. There are a number of addons which include precompiled static libs for iOS. At the least there will need to be a large, loud note in the CHANGELOG about recompiling old static libs for C++11 ... unless they work fine with the ld stdlib. I'm not 100% sure either way.

[danoli3]

@danomatika Yeah iOS libs use their own iOS SDK version of the i386 and x86_64 which is lipo'd into the static fat lib. This fat lib is all compiled with libc++ C++11 (i386, x86_64, armv7, arm64)

Definitely need to add to the changelog all the information about this for libc++ for iOS
Libc++ and libstdc++ are not compatible at all. For example the size of core types like strings are different and will usually be the first place of a crash or not able to link.
Currently the core re-uses the OS X i386 for some of the simulator libraries, however these are using the OSX SDK and not the iOS SDK (which can cause incompatibility issues (main one I've found is min iOS target issues)). So we will have to make sure developers realise they can't re-use these two, especially for 0.9+ with the i386 being libstdc++ for OS X and iOS being libc++ for i386.

All static addon libraries that include libs for the simulator will need to be recompiled anyway to support x86_64 as well so as long as they realise the libc++ for iOS this should be okay for both simulator libs.

Just to clarify:
iPhone 5 simulator in debug mode will be linked against the iOS i386
iPhone 5 simulator in release mode will be linked against the iOS x86_64 (by default, unless build active architecture is set to YES), then i386.

iPhone 6 simulator in debug/release mode will be linked against the iOS x86_64

[pizthewiz]

Currently the core re-uses the OS X i386 for some of the simulator libraries

Ahh interesting, it won't blow up statically linking to libs with different settings?

iPhone 5 simulator in release mode will be linked against the iOS x86_64 (by default, unless build active architecture is set to YES), then i386.

The iPhone 5 runs an Apple A7 ARMv7s so I'm pretty sure the simulator and device will only ever run i386 and ARMv7 libs respectively. The 64-bit devices/simulator can run 32-bit libs, but not the other way around, no?

[danoli3]

Yeah it would blow up, I've made changes to remove this problem in my
current 0.9.0 ios branch, by making it link only to iOS fat libs, I'll PR
that core changes today like the OSX PR.

And it is strange thing with the new Xcode 6 for release mode for using a
lib that should be using i386 will use x86_64 (by default).
When previously this was not the case.
Anyway that is fixed by build active Architechures to YES.

On Saturday, February 7, 2015, pizthewiz notifications@github.com wrote:

Currently the core re-uses the OS X i386 for some of the simulator
libraries

Ahh interesting, it won't blow up statically linking to libs with
different settings?

iPhone 5 simulator in release mode will be linked against the iOS x86_64
(by default, unless build active architecture is set to YES), then i386.

The iPhone 5 runs an Apple A7 ARMv7s so I'm pretty sure the simulator and
device will only ever run i386 and ARMv7 libs respectively. The 64-bit
devices/simulator can run 32-bit libs, but not the other way around, no?

—
Reply to this email directly or view it on GitHub
#2016 (comment)
.

[bilderbuchi]

closed by #3627, afaict.

[kylemcdonald]

:)

@danoli3 i noticed that https://github.com/danoli3/apothecary-den still shows "unknown" for port audio and rtAudio, is this actually a problem or just something that's out of date?

[ofTheo]

@kylemcdonald rtAudio has been built and updated for OS X.

[danoli3]

@kylemcdonald Updated apothecary-den to latest, all there now as well and passing

[kylemcdonald]

@danoli3 it looks like now poco says "error" and portaudio "unknown"... maybe just a glitch with travis timeouts for poco though?