Build afni.clean

[afni-rickr]

No description provided.

[yarikoptic]

@afni-rickr I have added a commit which adds setuptools installation on macos, lets see if it helps at all in that failing stage.

[yarikoptic]

ok -- we got further

but circle-ci seems to sensor outputs and I cannot understand what it wants for "ModuleNotFoundError: No module named '****py'"

[1804/1905] Linking C shared library targets_built/libplug_maxima.dylib
[1805/1905] Linking C shared library targets_built/libplug_scatplot.dylib
[1806/1905] Linking C executable targets_built/****_open
[1807/1905] Linking C shared library targets_built/libplug_3dsvm.dylib
[1808/1905] Linking C shared library targets_built/libplug_nlfit.dylib
[1809/1905] Linking C executable targets_built/Xphace
[1810/1905] Linking C executable targets_built/aiv
[1811/1905] Linking C executable targets_built/tokens
[1811/1905] cd /tmp/src/****/src/python_scripts && bash /tmp/src/****/cmake/make_sure_****py_is_installed.sh /usr/local/Frameworks/Python.framework/Versions/3.12/bin/python3.12
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named '****py'
Installing ****py...
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try brew install
    xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a Python library that isn't in Homebrew,
    use a virtual environment:
    
    python3 -m venv path/to/venv
    source path/to/venv/bin/activate
    python3 -m pip install xyz
    
    If you wish to install a Python application that isn't in Homebrew,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. You can install pipx with
    
    brew install pipx
    
    You may restore the old behavior of pip by passing
    the '--break-system-packages' flag to pip, or by adding
    'break-system-packages = true' to your pip.conf file. The latter
    will permanently disable this error.
    
    If you disable this error, we STRONGLY recommend that you additionally
    pass the '--user' flag to pip, or set 'user = true' in your pip.conf
    file. Failure to do this can result in a broken Homebrew installation.
    
    Read more about this behavior here: <https://peps.python.org/pep-0668/>

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
[1813/1905] Linking CXX executable targets_built/dcm2niix_****
FAILED: targets_built/dcm2niix_**** 
: && /Applications/Xcode-13.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -msse2 -mfpmath=sse -O3 -DNDEBUG -isysroot /Applications/Xcode-13.4.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names -Wl,-dead_strip -Wl,-stack_size -Wl,0x1000000    -Wl,-sectcreate,__TEXT,__info_plist,/tmp/src/****/Info.plist src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/main_console.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/nii_dicom.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/jpg_0XC3.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/ujpeg.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/nifti1_io_core.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/nii_foreign.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/nii_ortho.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/nii_dicom_batch.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/cJSON.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/base64.cpp.o -o targets_built/dcm2niix_****   && :
ld: can't open -sectcreate file: /tmp/src/****/Info.plist
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[1817/1905] Linking C executable targets_built/3dsvm
FAILED: src/python_scripts/CMakeFiles/dev_install_****py /tmp/src/****/build/src/python_scripts/CMakeFiles/dev_install_****py 
cd /tmp/src/****/src/python_scripts && bash /tmp/src/****/cmake/make_sure_****py_is_installed.sh /usr/local/Frameworks/Python.framework/Versions/3.12/bin/python3.12
ninja: build stopped: subcommand failed.

Let's see if I could fetch any logs "raw" using our https://github.com/con/tinuous since I see no button in circle ci web ui, uff.

edit: I found the button but that file has also ModuleNotFoundError: No module named '****py' ... brr

[mrneont]

From this bit of circleci text:

[1811/1905] cd /tmp/src/****/src/python_scripts && bash /tmp/src/****/cmake/make_sure_****py_is_installed.sh /usr/local/Frameworks/Python.framework/Versions/3.12/bin/python3.12
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named '****py'
Installing ****py...
�[1;31merror�[0m: �[1mexternally-managed-environment�[0m

... I think the error occurs when "make_sure_afnipy_is_installed.sh" tries to have Python run import afnipy.

It's further whining, some of which might be related to this, is:

�[31m�[0m This environment is externally managed
�[31m╰─>�[0m To install Python packages system-wide, try brew install
�[31m   �[0m xyz, where xyz is the package you are trying to
�[31m   �[0m install.
�[31m   �[0m 
�[31m   �[0m If you wish to install a Python library that isn't in Homebrew,
�[31m   �[0m use a virtual environment:
�[31m   �[0m 
�[31m   �[0m python3 -m venv path/to/venv
�[31m   �[0m source path/to/venv/bin/activate
�[31m   �[0m python3 -m pip install xyz
�[31m   �[0m 
�[31m   �[0m If you wish to install a Python application that isn't in Homebrew,
�[31m   �[0m it may be easiest to use 'pipx install xyz', which will manage a
�[31m   �[0m virtual environment for you. You can install pipx with
�[31m   �[0m 
�[31m   �[0m brew install pipx
�[31m   �[0m 
�[31m   �[0m You may restore the old behavior of pip by passing
�[31m   �[0m the '--break-system-packages' flag to pip, or by adding
�[31m   �[0m 'break-system-packages = true' to your pip.conf file. The latter
�[31m   �[0m will permanently disable this error.
�[31m   �[0m 
�[31m   �[0m If you disable this error, we STRONGLY recommend that you additionally
�[31m   �[0m pass the '--user' flag to pip, or set 'user = true' in your pip.conf
�[31m   �[0m file. Failure to do this can result in a broken Homebrew installation.
�[31m   �[0m 
�[31m   �[0m Read more about this behavior here: <https://peps.python.org/pep-0668/>

�[1;35mnote�[0m: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
�[1;36mhint�[0m: See PEP 668 for the detailed specification.
[1813/1905] Linking CXX executable targets_built/dcm2niix_****
FAILED: targets_built/dcm2niix_**** 
: && /Applications/Xcode-13.4.1.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -msse2 -mfpmath=sse -O3 -DNDEBUG -isysroot /Applications/Xcode-13.4.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX12.3.sdk -Wl,-search_paths_first -Wl,-headerpad_max_install_names -Wl,-dead_strip -Wl,-stack_size -Wl,0x1000000    -Wl,-sectcreate,__TEXT,__info_plist,/tmp/src/****/Info.plist src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/main_console.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/nii_dicom.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/jpg_0XC3.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/ujpeg.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/nifti1_io_core.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/nii_foreign.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/nii_ortho.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/nii_dicom_batch.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/cJSON.cpp.o src/crorden/dcm2niix_console/CMakeFiles/dcm2niix.dir/base64.cpp.o -o targets_built/dcm2niix_****   && :
ld: can't open -sectcreate file: /tmp/src/****/Info.plist
clang: error: linker command failed with exit code 1 (use -v to see invocation)
[1817/1905] Linking C executable targets_built/3dsvm
FAILED: src/python_scripts/CMakeFiles/dev_install_****py /tmp/src/****/build/src/python_scripts/CMakeFiles/dev_install_****py 
cd /tmp/src/****/src/python_scripts && bash /tmp/src/****/cmake/make_sure_****py_is_installed.sh /usr/local/Frameworks/Python.framework/Versions/3.12/bin/python3.12
ninja: build stopped: subcommand failed.

[yarikoptic]

might be, might be... might be worth also listening to it and just add that venv there , let's see...

[mrneont]

With macOS nowadays, we have had trouble nowadays getting Python dependencies into the brew-installed Python. They don't seem to like mixing brew-Python with pip-installed dependencies. For users, we pretty much recommend that people skip the brew-Python part and just do miniconda. That would be a bit of refactoring here, though. Will see how the venv insertion goes.

[yarikoptic]

I thought something broke annex again but it seems that the server at NIH is down?!

  'annexkey': 'MD5E-s10886--08fb764a4acc4dd765b37f3fabdcc879.HEAD',

  'error_message': 'Remote ****_ci_test_data not usable by git-annex; setting '

                   'annex-ignore\n'

                   'https://****.nimh.nih.gov/pub/dist/data/****_ci_test_data/.git/config '

                   'download failed: ConnectionFailure '

                   'Network.Socket.getAddrInfo (called with preferred socket '

                   'type/protocol: AddrInfo {addrFlags = [], addrFamily = '

                   'AF_UNSPEC, addrSocketType = Stream, addrProtocol = 6, '

                   'addrAddress = 0.0.0.0:0, addrCanonName = Nothing}, host '

                   'name: Just "****.nimh.nih.gov", service name: Just "443"): '

                   'does not exist (Temporary failure in name resolution)',

and locally I get

[2024-06-12 09:22:32.630900821] (Utility.Url) Request {
  host                 = "afni.nimh.nih.gov"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding","identity"),("User-Agent","git-annex/10.20240129")]
  path                 = "/pub/dist/data/afni_ci_test_data/.git/annex/objects/688/d89/MD5E-s10886--08fb764a4acc4dd765b37f3fabdcc879.HEAD/MD5E-s10886--08fb764a4acc4dd765b37f3fabdcc879.HEAD"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}

[mrneont]

I can log into the server, so I don't think it's down. I'm not sure what is causing those errors.

[yarikoptic]

you can login via ssh, it not a guarantee that webserver (apache2? nginx?) is running... what is its uptime command output btw since now it is back up and available. I will push rewrite of the last commit to retrigger CI

edit: I just dropped the tt temporary commit so we get cleaner history while at it

[mrneont]

@yarikoptic

If it turns out that Python dependencies don't install well with brew's python and the "venv" stuff, then we could go the miniconda route, such as this (if it's a different chip on circleci, we can adjust that part easily):

# Download and install (macOS, Intel chip)
curl -OL https://repo.continuum.io/miniconda/Miniconda3-latest-MacOSX-x86_64.sh
bash Miniconda3-latest-MacOSX-x86_64.sh -b

# Initialize conda in shells
~/miniconda3/bin/conda init bash tcsh zsh

# Update conda version (opt)
conda update -n base -c defaults conda

# Make env: AFNI minimal Python (for now, exclude: flask and flask-cors)
conda create -y                  \
    -n py3_afni_cci              \
    python                       \
    "matplotlib>=2.2.3"          \
    numpy                        \
    scipy
# ... and any more Python dependencies for CircleCI could be added to this list.
    
# activate env
conda activate py3_afni_cci

[yarikoptic]

the best would be for some afni folk with a mac to just give it a short following those commands in .circleci -- anyone with an aluminum case?

NB no need even for a dedicated environment, just use base one of conda

[yarikoptic]

the best would be for some afni folk with a mac to just give it a short following those commands in .circleci -- anyone with an aluminum case?

I guess nobody... ok, let's continue somehow...

[mrneont]

Hi, Yarik-

Sorry for being slow on this. Preparation from OHBM is happening, as well as a backlog of other things.

[yarikoptic]

FTR: here is a useful howto debug via ssh on circleci (didn't try yet) https://circleci.com/docs/ssh-access-jobs/#overview

[yarikoptic]

there seems to be a cocktail of python installations etc there.

==> Running `brew cleanup python@3.12`...
==> Pouring python-setuptools--70.0.0.monterey.bottle.tar.gz
🍺  /usr/local/Cellar/python-setuptools/70.0.0: 272 files, 3MB
==> Running `brew cleanup python-setuptools`...
==> Upgrading 1 dependent of upgraded formulae:
Disable this behaviour by setting HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
python@3.9 3.9.13_1 -> 3.9.19
....
        the python interpreter:
        /tmp/src/****/build/venv/bin/python3.12

Let's just install setuptools within the venv...

[afni-rickr]

It's all green! @yarikoptic, thank you so much! I had indeed been trying to reproduce this locally, but it seems like we will get bitten when this is updated to use newer systems. But that is for another day... :) Thanks again!

[yarikoptic]

yeay, we did it! if only I was more careful and did not do a stupid thing, we were done days ago! ;-)

now please just keep it green!