Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

OS X install #41

Open
andypiper opened this Issue · 28 comments

9 participants

@andypiper

currently, install.sh is Linux / Ubuntu & Debian specific

Refer to this discussion on LinkedIn for initial thoughts on this subject.

Also see the homebrew docs on Python

Attempting to find homebrew equivalent packages for some of the dependencies results in:

for i in bzr gfortran haproxy  \
for>     ruby-sass libatlas-dev libatlas3gf-base libblas3gf \
for>     libevent-dev libgfortran3 liblapack-dev liblapack3gf \
for>     libpq-dev libyaml-dev libxml2-dev libxslt1-dev libumfpack5.4.0 \
for>     openssl python2.7-dev python-m2crypto python-numpy python-pip \
for>     python-scipy python-zdaemon swig uuid-dev uuid-runtime
for> do
for> brew search $i
for> done
bzr
gfortran
haproxy
No formula found for "ruby-sass". Searching open pull requests...
No formula found for "libatlas-dev". Searching open pull requests...
No formula found for "libatlas3gf-base". Searching open pull requests...
No formula found for "libblas3gf". Searching open pull requests...
No formula found for "libevent-dev". Searching open pull requests...
No formula found for "libgfortran3". Searching open pull requests...
No formula found for "liblapack-dev". Searching open pull requests...
No formula found for "liblapack3gf". Searching open pull requests...
No formula found for "libpq-dev". Searching open pull requests...
No formula found for "libyaml-dev". Searching open pull requests...
No formula found for "libxml2-dev". Searching open pull requests...
No formula found for "libxslt1-dev". Searching open pull requests...
No formula found for "libumfpack5.4.0". Searching open pull requests...
openssl
No formula found for "python2.7-dev". Searching open pull requests...
No formula found for "python-m2crypto". Searching open pull requests...
No formula found for "python-numpy". Searching open pull requests...
No formula found for "python-pip". Searching open pull requests...
No formula found for "python-scipy". Searching open pull requests...
No formula found for "python-zdaemon". Searching open pull requests...
swig
No formula found for "uuid-dev". Searching open pull requests...
No formula found for "uuid-runtime". Searching open pull requests...

However it is possible to pip install m2crypto, numpy, scipy and zdaemon AS LONG AS you install swig and gfortran first.

Of the libs, I can see there's libyaml in brew and packages called ossp-uuid and libpqxx but I do not know whether they match the other requirements.

@Adniel

Partly I guess the missing dependencies are named differently. I searched macport and found some of the missing packages, but with slightly different naming.

Another option for actually installing zato on mac would be to extend the current buildout with source builds for the missing dependencies. That is cumbersome but probably possible. If I find time I will give it a try.

@michaelmior

Note that Homebrew doesn't have any -dev packages, but development headers are always installed with the corresponding packages. Also, any language-specific packages are not managed by Homebrew (e.g. ruby-sass, python-m2crypto, etc.)

@sanumala

If Homebrew doesn't fit can we think of any alternate tools/build systems? Personally I would not suggest source builds for the missing dependencies since it make zato more heavy and maintenance of such third party sources will be cumbersome.

@michaelmior

I think Homebrew is probably still a good fit. You can still depend on Python and Ruby packages, they will just be installed via gem or pip and don't have their own package in Homebrew.

@dsuch
Collaborator

Indeed, Michael. What install.sh really does it to pave way for virtualenv, pip and buildout.

https://github.com/zatosource/zato/blob/master/code/install.sh
https://github.com/zatosource/zato/blob/master/code/buildout.cfg
https://github.com/zatosource/zato/blob/master/code/versions.cfg

There are several dozen of Python dependencies for Zato and buildout takes care of installing them all. It's just that some of them need OS-level packages to be installed first, things like C headers/lib to compile with/against or similar stuff.

So the majority of the work is done by buildout.

The way I originally came up with a list of DEB packages to install was to issue ..

sudo pip install --upgrade distribute
sudo pip install --upgrade virtualenv

virtualenv .

./bin/python bootstrap.py -v 1.7.0
./bin/buildout

.. and then for each dependency from versions.cfg failing to install I somehow had to figure out what system-level packages were needed. Perhaps the same approach will work here?

@michaelmior

That would be a good start. Although a note that anything using sudo is highly discouraged in Homebrew. Everything should be installed into /usr/local which the user will have permission to write to if Homebrew is configured correctly.

Also, zato currently uses some libraries that are unavailable on Mac OS X such as inotify.

@michaelmior

You may want to check out watchdog which provides cross-platform support for file monitoring.

@dsuch
Collaborator

Thanks Michael, in fact Zato can already use watchdog :-)

https://github.com/zatosource/zato/tree/master/code/zato-server/src/zato/server/pickup

This is used for hot-deployment

https://zato.io/docs/admin/guide/installing-services.html#hot-deploying-from-command-line

If you have a look at the code, you'll note that it first attempts to use gevent_inotifyx and then falls back to watchdog. So watchdog as such can and will be used but the question is, how to make buildout forget about inotifyx if platform isn't Linux - will look it up.

@michaelmior

Great :) Any reason that inotify can't just be ditched in favour of watchdog if watchdog already works?

@dsuch
Collaborator

Well, I'd just need to find out if watchdog and gevent are any friends. I don't know how watchdog works really, is it safe to use them together. A watchdog must time.sleep() from time to time which means the current thread is blocked. But with gevent this may not be so simple, the current thread can be the main server thread so we don't want it to sleep :-)

Hence my idea was to use async inotifyx on Linux (where gevent.sleep can be used and I'm sure it won't disturb the main server thread) and watchdog elsewhere.

@andypiper

Agree that Homebrew is potentially still a good fit, I was just aiming to get this discussion started here on Github (rather than on LinkedIn!).

@andypiper
[zato-1.0] python bootstrap.py
Downloading http://pypi.python.org/packages/2.7/s/setuptools/setuptools-0.6c11-py2.7.egg
zc.buildout 2 needs distribute, not setuptools.  Are you using an outdated bootstrap.py?  Make sure you have the latest version downloaded from http://downloads.buildout.org/2/bootstrap.py
@dsuch
Collaborator

Wrt installing things optionally depending on conditions out of buildout's influence, I now recall this was a major pain.

This is why Oracle and WebSphere MQ have to be enabled manually

https://zato.io/docs/admin/guide/enabling-extra-libs.html

It seemed quite a task to make buildout be aware that MQ or Oracle's libs and headers were in such and such places only if such and such env variables were exported etc.

My point is, if you guys can make Zato install on Mac through any means, I'll be happy to integrate the work into install.sh and buildout somehow. Let the fact that there is still buildout itself and some optional platform-specific libraries to think of doesn't discourage you in any way, please.

So if resolving this issue leads to some sort of a install-mac.sh script, I'll be very happy to merge it seamlessly into the rest of the installer so for instance, install.sh will call install-deb.sh or install-mac.sh depending on where it's on. With time, an install-rpm.sh will be added.

I also hope that Linux packagers will want to make Zato installable through 'apt-get install zato' in some future but that's a different story still.

@andypiper

[13:08:31] currently stuck at the inotifyx dependency on OS X build
[13:09:09] I've worked out you need to brew install swig, then gfortran, then run the pip installs of modules mentioned in install.sh, gem install sass, run bootstrap… and then buildout fails at inotifyx

@gregturn

Have you entertained adding homebrew recipes for the cases where the package doesn't yet have a recipe? It might be a bigger cost upfront, but releasing it to homebrew potentially takes the burden off of you to maintain it in the future.

@dsuch
Collaborator

OK, I finally have (remote) access to a Mac box and will now engage more in this issue. This is a priority and I'm putting all other features aside till it's ready.

@dsuch
Collaborator

I'm not sure if you're working on it, @sanumala, but it seems I've got it working to a good extent, i.e. it installs with no problems and all tests pass.

No new recipes were needed, it was just a matter of finding the correct formulas or adding few missing dependencies with pip. Some formulas have to be tapped but I understand this is acceptable.

Judging from what people do elsewhere, pip is what is to be used for installing M2Crypto, for instance.

It's a bit messy right now so I'll clean it up and will hopefully upload it tomorrow for you guys to try out..

@dsuch
Collaborator

Can you please try the following?

$ wget https://zato.io/download/zato-1.0.tar.bz2 .
$ tar xjvf zato-1.0.tar.bz2 && cd zato-1.0
$ mv install.sh install.sh.bak
$ mv buildout.cfg buildout.cfg.bak
$ mv versions.cfg versions.cfg.bak
$ curl -s https://gist.github.com/dsuch/5653464/raw -o buildout.cfg
$ curl -s https://gist.github.com/dsuch/5653471/raw -o versions.cfg
$ curl -s https://gist.github.com/dsuch/5653474/raw -o install.sh
$ chmod u+x install.sh
$ ./install.sh
[snip output]
OK
$ ./bin/zato --version
Zato 1.0
$

This will be incorporated in 1.1 but for now this is how installation on OS X will look like. In the next release it will be only a matter of running ./install.sh and everything will be taken care of.

I'm leaving turning it into a proper homebrew recipe to future contributors. This should be rather simple, all the dependencies are pin-pointed so it's really only a matter of authoring a bit of Ruby code and pushing it to GH.

Thanks!

@andypiper

Works :+1:

However, a few notes:

  • personally I use the git binary installed by the Github app on OS X, so I didn't want brew to install git. Some dependency checking might be useful. On the other hand, a brew formula is probably just going to pre-req git so I'm wrong anyway :-)
  • I had previously pip install'ed the dependencies and they ended up in /Library/Python/2.7/site-packages and not in the brew prefix location -> the symlink step needed to be changed
  • the final command should be ./bin/zato --version :-)
@dsuch
Collaborator

OK, I updated the final command :-)

I also tagged this issue with a 'contributor-needed' flag. Right now, given that we know it works, it hopefully should really be simple for someone actually familiar with OS X to come up with a homebrew formula.

So I'm not closing it and if there's anyone who can help - please join in Thanks!

@dsuch
Collaborator

Hi there, can you please tell me what your OS X version is?

I'm testing everything on 10.8 and have absolutely no problems but a user reported privately he cannot install Zato 1.1 on OS X 10.6 - this is the only difference I can see.

Can you tell what your versions is? Thanks!

@dsuch
Collaborator

Eh, I forgot to mention - the issue is with M2Crypto - pip just cannot find libssl even though brew install openssl pulls everything nicely.

@trebor74hr

M2Crypto and openssl problems

Installing openssl with:

brew install openssl

will report following:

Note that the libraries built tend to be 32-bit only, even on Snow Leopard.
This formula is keg-only: so it was not symlinked into /usr/local.
Mac OS X already provides this software and installing another version in
parallel can cause all kinds of trouble.

installing m2crypto with pip will report:

> pip install m2crypto
...
ld: library not found for -lssl
collect2: ld returned 1 exit status
error: command 'llvm-gcc-4.2' failed with exit status 1

Solution 1 - install prebuilt egg

There are prebuilt eggs listed here, and the one that suites me is:

easy_install http://chandlerproject.org/pub/Projects/MeTooCrypto/M2Crypto-0.21.1-py2.7-macosx-10.7-intel.egg

(strange thing, this command will install egg and pull source of m2crypto and build it again - and fail ;). Since egg is installed normally, everything ends ok).

Testing zato:

zato-1.1$ . bin/activate
zato-1.1$ bin/zato --version
Zato 1.1

Solution 2 - install from source with --openssl option

It is possible that the issue is that m2crypto 0.21.1 tries to use openssl prebuilt in OSX (0,9.8r) which is somehow not compatible with version 0.21.1 (btw. openssl 1.0 did some api changes that are not backward compatible).

M2crypto has option openssl to tell where to locate openssl installation. Brew installs this (in my case) in:

/usr/local/Cellar/openssl/1.0.1c/

Procedure is - dowload source, extract and custom build+install:

m2crypto$ python setup.py build_ext --openssl=/usr/local/Cellar/openssl/1.0.1c/

m2crypto$ python setup.py install
@baocaixiong

When I installed on my mac os, it is always tell me:

Getting distribution for 'zdaemon==2.0.4'.
Got zdaemon 2.0.4.
Getting distribution for 'setproctitle==1.1.6'.
In file included from src/setproctitle.c:16:
In file included from src/spt_status.h:13:
src/c.h:33:22: error: expected parameter declarator
HIDDEN extern size_t strlcpy(char *dst, const char *src, size_t siz);
                     ^
/usr/include/secure/_string.h:105:44: note: expanded from macro 'strlcpy'
  __builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
                                           ^
/usr/include/secure/_common.h:39:62: note: expanded from macro '__darwin_obsz'
#define __darwin_obsz(object) __builtin_object_size (object, _USE_FORTIFY_LEVEL > 1 ? 1 : 0)
                                                             ^
/usr/include/secure/_common.h:30:32: note: expanded from macro '_USE_FORTIFY_LEVEL'
#    define _USE_FORTIFY_LEVEL 2
                               ^
In file included from src/setproctitle.c:16:
In file included from src/spt_status.h:13:
src/c.h:33:22: error: expected ')'
/usr/include/secure/_string.h:105:44: note: expanded from macro 'strlcpy'
  __builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
                                           ^
/usr/include/secure/_common.h:39:62: note: expanded from macro '__darwin_obsz'
#define __darwin_obsz(object) __builtin_object_size (object, _USE_FORTIFY_LEVEL > 1 ? 1 : 0)
                                                             ^
/usr/include/secure/_common.h:30:32: note: expanded from macro '_USE_FORTIFY_LEVEL'
#    define _USE_FORTIFY_LEVEL 2
                               ^
src/c.h:33:22: note: to match this '('
/usr/include/secure/_string.h:105:44: note: expanded from macro 'strlcpy'
  __builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
                                           ^
/usr/include/secure/_common.h:39:53: note: expanded from macro '__darwin_obsz'
#define __darwin_obsz(object) __builtin_object_size (object, _USE_FORTIFY_LEVEL > 1 ? 1 : 0)
                                                    ^
In file included from src/setproctitle.c:16:
In file included from src/spt_status.h:13:
src/c.h:33:22: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
HIDDEN extern size_t strlcpy(char *dst, const char *src, size_t siz);
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/secure/_string.h:105:44: note: expanded from macro 'strlcpy'
  __builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
                                           ^~~~~~~~~~~~~~~~~~~~
/usr/include/secure/_common.h:39:31: note: expanded from macro '__darwin_obsz'
#define __darwin_obsz(object) __builtin_object_size (object, _USE_FORTIFY_LEVEL > 1 ? 1 : 0)
                              ^~~~~~~~~~~~~~~~~~~~~
In file included from src/setproctitle.c:16:
In file included from src/spt_status.h:13:
src/c.h:33:22: error: conflicting types for '__builtin___strlcpy_chk'
/usr/include/secure/_string.h:105:3: note: expanded from macro 'strlcpy'
  __builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
  ^
src/c.h:33:22: note: '__builtin___strlcpy_chk' is a builtin with type 'unsigned long (char *, const char *, unsigned
      long, unsigned long)'
/usr/include/secure/_string.h:105:3: note: expanded from macro 'strlcpy'
  __builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
  ^
1 warning and 3 errors generated.
error: Setup script exited with error: command 'gcc' failed with exit status 1
An error occurred when trying to install setproctitle 1.1.6. Look above this message for any errors that were output by easy_install.
While:
  Installing zato.
  Getting distribution for 'setproctitle==1.1.6'.
Error: Couldn't install: setproctitle 1.1.6

OK

i hope someone can help me.
thanks.
--baocaixiong

@michaelmior

Try CFLAGS='-D_FORTIFY_SOURCE=0' pip install setproctitle==1.1.6 before installing Zato.

@baocaixiong

step 1: CFLAGS='-D_FORTIFY_SOURCE=0' pip install setproctitle==1.1.6

step2: ./install.sh

Got sec-wall 1.2.
Getting distribution for 'setproctitle==1.1.6'.
In file included from src/setproctitle.c:16:
In file included from src/spt_status.h:13:
src/c.h:33:22: error: expected parameter declarator
HIDDEN extern size_t strlcpy(char *dst, const char *src, size_t siz);
                     ^
/usr/include/secure/_string.h:105:44: note: expanded from macro 'strlcpy'
  __builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
                                           ^
/usr/include/secure/_common.h:39:62: note: expanded from macro '__darwin_obsz'
#define __darwin_obsz(object) __builtin_object_size (object, _USE_FORTIFY_LEVEL > 1 ? 1 : 0)
                                                             ^
/usr/include/secure/_common.h:30:32: note: expanded from macro '_USE_FORTIFY_LEVEL'
#    define _USE_FORTIFY_LEVEL 2
                               ^
In file included from src/setproctitle.c:16:
In file included from src/spt_status.h:13:
src/c.h:33:22: error: expected ')'
/usr/include/secure/_string.h:105:44: note: expanded from macro 'strlcpy'
  __builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
                                           ^
/usr/include/secure/_common.h:39:62: note: expanded from macro '__darwin_obsz'
#define __darwin_obsz(object) __builtin_object_size (object, _USE_FORTIFY_LEVEL > 1 ? 1 : 0)
                                                             ^
/usr/include/secure/_common.h:30:32: note: expanded from macro '_USE_FORTIFY_LEVEL'
#    define _USE_FORTIFY_LEVEL 2
                               ^
src/c.h:33:22: note: to match this '('
/usr/include/secure/_string.h:105:44: note: expanded from macro 'strlcpy'
  __builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
                                           ^
/usr/include/secure/_common.h:39:53: note: expanded from macro '__darwin_obsz'
#define __darwin_obsz(object) __builtin_object_size (object, _USE_FORTIFY_LEVEL > 1 ? 1 : 0)
                                                    ^
In file included from src/setproctitle.c:16:
In file included from src/spt_status.h:13:
src/c.h:33:22: warning: type specifier missing, defaults to 'int' [-Wimplicit-int]
HIDDEN extern size_t strlcpy(char *dst, const char *src, size_t siz);
                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/secure/_string.h:105:44: note: expanded from macro 'strlcpy'
  __builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
                                           ^~~~~~~~~~~~~~~~~~~~
/usr/include/secure/_common.h:39:31: note: expanded from macro '__darwin_obsz'
#define __darwin_obsz(object) __builtin_object_size (object, _USE_FORTIFY_LEVEL > 1 ? 1 : 0)
                              ^~~~~~~~~~~~~~~~~~~~~
In file included from src/setproctitle.c:16:
In file included from src/spt_status.h:13:
src/c.h:33:22: error: conflicting types for '__builtin___strlcpy_chk'
/usr/include/secure/_string.h:105:3: note: expanded from macro 'strlcpy'
  __builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
  ^
src/c.h:33:22: note: '__builtin___strlcpy_chk' is a builtin with type 'unsigned long (char *, const char *, unsigned
      long, unsigned long)'
/usr/include/secure/_string.h:105:3: note: expanded from macro 'strlcpy'
  __builtin___strlcpy_chk (dest, src, len, __darwin_obsz (dest))
  ^
1 warning and 3 errors generated.
error: Setup script exited with error: command 'clang' failed with exit status 1
An error occurred when trying to install setproctitle 1.1.6. Look above this message for any errors that were output by easy_install.
While:
  Installing zato.
  Getting distribution for 'setproctitle==1.1.6'.
Error: Couldn't install: setproctitle 1.1.6

OK
@hachpai

Same issue.
I had to do:
CFLAGS='-D_FORTIFY_SOURCE=0' ./install.sh
Then I had a version conflict:

Got zdaemon 2.0.4.
While:
  Installing zato.
Error: There is a version conflict.
We already have: circuits 1.51.5

OK

So i've done:
pip install circuits

And then change circuits = 1.5 by circuits = 2.1 in versions.cfg.

After a last CFLAGS='-D_FORTIFY_SOURCE=0' ./install.sh everything worked fine.

Hope it'll help. It seems to be a issue with osX maverick...

@dsuch
Collaborator

Hello,

the recommended way to install Zato for development under OS X will be to use the Vagrantfile

Please check it out here if you'd like to confirm it before it's added to the installation docs

https://mailman-mail5.webfaction.com/pipermail/zato-discuss/2014-April/000360.html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.