Running Petrel on localhost OS X

[mihneagiurgea]

I can't run the sample wordcount topology on my Mac OS X 10.7.4 localhost because of the following error:

seraph:~/ubercode/Petrel/samples/wordcount$ ./buildandrun --config topology.yaml
2312 [Thread-7] INFO  backtype.storm.daemon.worker ... has finished loading
....
setup_splitsentence.sh: line 47: flock: command not found
setup_splitsentence.sh: line 47: flock: command not found
setup_splitsentence.sh: line 47: flock: command not found

The "flock" utility is missing from OS X, and I can't find the setup_splitsentence.sh file that is using it.

Link to a more complete stack-trace: http://pastie.org/5718499

[barrywhart]

The setup_*.sh scripts are created from the string template in the intercept() function of petrel/petrel/package.py.

Petrel uses the "flock" command in order to create a single virtualenv per Storm worker machine and share it across all the processes from that topology.

It appears the "flock" command is not available on Mac OS X:

http://stackoverflow.com/questions/10526651/mac-os-x-equivalent-of-linux-flock1-command

I have made some experimental changes to package.py and attached a copy for you to try. The changes begin at the comment:

On Mac OS X, the "flock" command is not available

Please test these changes and let me know if they work. I'll get them in the Petrel trunk for you.

With these changes, each job on OS X will always create a new virtualenv. This will make jobs start up slower and use more disk space, but it should work. For your production environment, you will probably want to use an operating system which supports the "flock" command.

[barrywhart]

Here's a patch file with the changes mentioned in the previous comment - I think github.com discarded my email attachment.

diff --git a/petrel/petrel/package.py b/petrel/petrel/package.py
index a4a0a12..22b136a 100644
--- a/petrel/petrel/package.py
+++ b/petrel/petrel/package.py
@@ -202,17 +202,28 @@ shlock()            { _lock s; }   # obtain a shared lock
 unlock()            { _lock u; }   # drop a lock

 if [ $CREATE_VENV -ne 0 ]; then
-    if [ -d $VENV ];then
-        echo "Using existing venv: $VENV" >>$LOG 2>&1
-        shlock
-        source $VENV/bin/activate >>$LOG 2>&1
-        unlock
-    elif ! exlock_now;then
-        echo "Using existing venv: $VENV" >>$LOG 2>&1
-        shlock
-        source $VENV/bin/activate >>$LOG 2>&1
-        unlock
-    else
+    set +e
+    which flock
+    has_flock=$?
+    set -e
+    create_new=1
+    if [ "$has_flock" -eq "0" ]
+    then 
+        if [ -d $VENV ];then
+            echo "Using existing venv: $VENV" >>$LOG 2>&1
+            shlock
+            source $VENV/bin/activate >>$LOG 2>&1
+            unlock
+            create_new=0
+        elif ! exlock_now;then
+            echo "Using existing venv: $VENV" >>$LOG 2>&1
+            shlock
+            source $VENV/bin/activate >>$LOG 2>&1
+            unlock
+            create_new=0
+        fi
+    if [ "$create_new" -eq "1" ]
+    then
         echo "Creating new venv: $VENV" >>$LOG 2>&1
         virtualenv --system-site-packages --python python$PYVER $VENV >>$VENV_LOG 2>&1
         source $VENV/bin/activate >>$VENV_LOG 2>&1

[mihneagiurgea]

The patch you supplied did not work, the flock command is still called.

As far as I can tell, shlock is called when has_flock == 0, which in turn flock (as defined on line 201) - is this correct?

[barrywhart]

I want it to skip the code in this section:

if [ "$has_flock" -eq "0" ]
then

if "flock" is not available on the machine. When "flock" is missing it should always enter this "if":

if [ "$create_new" -eq "1" ]
then

I will post an updated patch with a couple of additional fixes. It may still not work but it should be pretty close.

I'm pretty busy with some other projects this week, so I don't have time to test the patches before sending them to you. Sorry! With these patches as a starting point, can you try to get it working and send me an updated patch when it is working for you?

[barrywhart]

diff --git a/petrel/petrel/package.py b/petrel/petrel/package.py
index a4a0a12..d239c11 100644
--- a/petrel/petrel/package.py
+++ b/petrel/petrel/package.py
@@ -202,17 +202,30 @@ shlock()            { _lock s; }   # obtain a shared lock
 unlock()            { _lock u; }   # drop a lock

 if [ $CREATE_VENV -ne 0 ]; then
-    if [ -d $VENV ];then
-        echo "Using existing venv: $VENV" >>$LOG 2>&1
-        shlock
-        source $VENV/bin/activate >>$LOG 2>&1
-        unlock
-    elif ! exlock_now;then
-        echo "Using existing venv: $VENV" >>$LOG 2>&1
-        shlock
-        source $VENV/bin/activate >>$LOG 2>&1
-        unlock
-    else
+    # On Mac OS X, the "flock" command is not available
+    set +e
+    which flock
+    has_flock=$?
+    set -e
+    create_new=1
+    if [ "$has_flock" -eq "0" ]
+    then 
+        if [ -d $VENV ];then
+            echo "Using existing venv: $VENV" >>$LOG 2>&1
+            shlock
+            source $VENV/bin/activate >>$LOG 2>&1
+            unlock
+            create_new=0
+        elif ! exlock_now;then
+            echo "Using existing venv: $VENV" >>$LOG 2>&1
+            shlock
+            source $VENV/bin/activate >>$LOG 2>&1
+            unlock
+            create_new=0
+        fi
+    fi
+    if [ "$create_new" -eq "1" ]
+    then
         echo "Creating new venv: $VENV" >>$LOG 2>&1
         virtualenv --system-site-packages --python python$PYVER $VENV >>$VENV_LOG 2>&1
         source $VENV/bin/activate >>$VENV_LOG 2>&1
@@ -231,7 +244,10 @@ if [ $CREATE_VENV -ne 0 ]; then
         if [ -f ./setup.sh ]; then
             /bin/bash ./setup.sh $CREATE_VENV >>$VENV_LOG 2>&1
         fi
-        unlock
+        if [ "$has_flock" -eq "0" ]
+        then 
+            unlock
+        fi
     fi
 else
     # This is a prototype feature where the topology specifies a virtualenv

[barrywhart]

Closing due to no response. Please open a new ticket if you can help test this on a Mac.

[barrywhart]

Fixed by pull request #20