New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix automatic IPP controller creation on OS X #206

Closed
annawoodard opened this Issue Apr 17, 2018 · 1 comment

Comments

Projects
None yet
1 participant
@annawoodard
Collaborator

annawoodard commented Apr 17, 2018

Automatic creation of the IPP controller fails on Mac. Here is a test which reproduces the problem:

(parsl_py36) [Anna-Woodards-MacBook-Pro] ~/ci/parsl/parsl/tests/test_data >python test_file_ipp.py
sites [{'site': 'Local_IPP', 'auth': {'channel': None}, 'execution': {'executor': 'ipp', 'provider': 'local', 'block': {'initBlocks': 4}}}]
site Local_IPP
auth {'channel': None}
channel None
execution {'executor': 'ipp', 'provider': 'local', 'block': {'initBlocks': 4}}
executor ipp
provider local
block {'initBlocks': 4}
initBlocks 4
globals {'lazyErrors': True}
lazyErrors True
site Local_IPP
auth {'channel': None}
channel None
execution {'executor': 'ipp', 'provider': 'local', 'block': {'initBlocks': 4}}
executor ipp
provider local
block {'initBlocks': 4}
initBlocks 4
SSH_AGENT_PID 676
TERM_PROGRAM iTerm.app
TERM xterm-256color
SHELL /bin/bash
HISTSIZE 800000
TMPDIR /var/folders/nt/sny5t5bd3js4y0fwhpbv32p00000gn/T/
Apple_PubSub_Socket_Render /private/tmp/com.apple.launchd.rfUzAYzuml/Render
TERM_PROGRAM_VERSION 3.1.20171019-nightly
TERM_SESSION_ID w1t7p0:B2D31661-E89E-4EC4-B759-C3E7B02EB5E0
USER awoodard
LD_LIBRARY_PATH :/Users/awoodard/local/lib
COMMAND_MODE unix2003
SSH_AUTH_SOCK /var/folders/nt/sny5t5bd3js4y0fwhpbv32p00000gn/T//ssh-fF1v22munma8/agent.675
__CF_USER_TEXT_ENCODING 0x1F5:0x0:0x0
LSCOLORS exfxcxdxbxegedabagacad
PATH /opt/local/bin:/opt/local/sbin:/Users/awoodard/software/anaconda3/envs/parsl_py36/bin:/Users/awoodard/software/anaconda3/bin:/Users/awoodard/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/git/bin:/Library/TeX/texbin:/usr/local/opt/fzf/bin:/usr/local/texlive/2016/bin/x86_64-darwin/
CONDA_PATH_BACKUP /Users/awoodard/software/anaconda3/bin:/Users/awoodard/.local/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin:/usr/local/git/bin:/Library/TeX/texbin:/usr/local/opt/fzf/bin:/usr/local/texlive/2016/bin/x86_64-darwin/
CONDA_PREFIX /Users/awoodard/software/anaconda3/envs/parsl_py36
PWD /Users/awoodard/ci/parsl/parsl/tests/test_data
LANG en_US.utf-8
PYTHONSTARTUP /Users/awoodard/.pystartup
ITERM_PROFILE dark colors
XPC_FLAGS 0x0
PS1 (parsl_py36) [\h] \w >
XPC_SERVICE_NAME 0
PARSL_TESTING True
SHLVL 1
HOME /Users/awoodard
COLORFGBG 15;0
CONDA_PS1_BACKUP [\h] \w >
ITERM_SESSION_ID w1t7p0:B2D31661-E89E-4EC4-B759-C3E7B02EB5E0
LOGNAME awoodard
LC_CTYPE en_US.UTF-8
FZF_DEFAULT_COMMAND rg --files --hidden --smartcase --glob "!.git/*"
CONDA_DEFAULT_ENV parsl_py36
DISPLAY /private/tmp/com.apple.launchd.0fyi8nxfEK/org.macosforge.xquartz:0
SECURITYSESSIONID 186a8
HISTFILE /Users/awoodard/.bash_history
COLORTERM truecolor
OLDPWD /Users/awoodard/ci/parsl/parsl/tests
_ /Users/awoodard/software/anaconda3/envs/parsl_py36/bin/python
DEBUG:parsl.executors.ipp:Starting IpyParallelExecutor with provider:<Local Execution Provider for site:Local_IPP>
DEBUG:parsl.executors.ipp:Launched block : 0:51031
DEBUG:parsl.executors.ipp:Launched block : 1:51032
DEBUG:parsl.executors.ipp:Launched block : 2:51034
DEBUG:parsl.executors.ipp:Launched block : 3:51036
DEBUG:parsl.executors.ipp:Starting executor
DEBUG:parsl.execution_provider.provider_factory:Created executor : <IPP Executor for site:Local_IPP>
DEBUG:parsl.dataflow.strategy:Scaling strategy: simple
DEBUG:parsl.dataflow.dflow:Using executors: {'Local_IPP': <IPP Executor for site:Local_IPP>}
INFO:parsl.dataflow.dflow:Task 0 submitted for App remote_side_bash_executor, waiting on tasks []
INFO:parsl.dataflow.dflow:Task 0 launched on site Local_IPP
DEBUG:parsl.dataflow.dflow:Task 0 launched with AppFut:<AppFuture at 0x10ca7e5c0 state=pending>
DEBUG:parsl.app.futures:Creating DataFuture with parent : <AppFuture at 0x10ca7e5c0 state=pending>
DEBUG:parsl.app.futures:Filepath : cat_out.txt
DEBUG:parsl.dataflow.strategy:Min:0 initBlocks:4 Max:10
DEBUG:parsl.dataflow.strategy:Tasks:1 Slots:4 Parallelism:0.75
@annawoodard

This comment has been minimized.

Collaborator

annawoodard commented Apr 17, 2018

@yadudoc and I had a look at this, it seems there are two issues:

  1. "&>>" syntax for append with redirect fails in bash 3
  2. export CORES=$(grep -c ^processor /proc/cpuinfo) does not work on OS X

annawoodard added a commit that referenced this issue Apr 17, 2018

annawoodard added a commit to Parsl/libsubmit that referenced this issue Apr 17, 2018

Get number of cores from `getconf`
This should work with both OS X and Linux. Partially fixes
Parsl/parsl#206.

@yadudoc yadudoc closed this in #208 Apr 17, 2018

yadudoc added a commit that referenced this issue Apr 17, 2018

Merge pull request #208 from Parsl/fix-mac-ipp-controller-creation-#206
Use bash 3-compliant syntax for append + redirect

@annawoodard annawoodard self-assigned this May 8, 2018

benclifford pushed a commit that referenced this issue Aug 9, 2018

Get number of cores from `getconf`
This should work with both OS X and Linux. Partially fixes
#206.

annawoodard pushed a commit that referenced this issue Sep 24, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment