Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download libs script update (adds #7907 wget2) - params and fixes deploy logs #8002

Merged
merged 34 commits into from
May 30, 2024

Conversation

danoli3
Copy link
Member

@danoli3 danoli3 commented May 27, 2024

Cherry picks: @dimitre #7907 (add wget2 and parallel downloads)

Further Updated the downloader script to support changes for parallel downloads

  • Added params to downloader script to support no ssl, multiple downloads in list
  • Validates download type
  • New Param: -k / --no-ssl (can be used to bypass SSL encryption if network or router or something MITM - use cautiously )
  • Silent param: -s
  • Fixed override to remove just target download type libraries (not all types as was before)
  • Fixed logging more clear as to what is happening
  • Validated wget2 / curl / wget working

Cleaned up download_libs with better logs and overwrite settings to the target download type

dimitre and others added 4 commits May 27, 2024 13:47
(cherry picked from commit 6e8819c)
…no ssl, wget2, curl, wget

Cleaned up download_libs with better logs and overwrite settings to the target download type
@danoli3 danoli3 requested a review from dimitre May 27, 2024 07:31
@dimitre
Copy link
Member

dimitre commented May 27, 2024

Hello @danoli3 I'm testing now in osx and wget2 calls are being called once for each download.
the whole point of using wget2 is download all three urls as a unique parameter, so they are being all downloaded simultaneously
echoing commands

 [downloader] ... 
Downloading [openFrameworksLibs_bleeding_osx_1.tar.bz2] @ [https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_1.tar.bz2]
/opt/homebrew/bin/wget2
wget2 --progress=bar https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_1.tar.bz2
Downloading [openFrameworksLibs_bleeding_osx_2.tar.bz2] @ [https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_2.tar.bz2]
/opt/homebrew/bin/wget2
wget2 --progress=bar https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_2.tar.bz2
Downloading [openFrameworksLibs_bleeding_osx_3.tar.bz2] @ [https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2]
/opt/homebrew/bin/wget2
wget2 --progress=bar https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2

EDIT: in the other PR

openFrameworksLibs_b  11% [============>                                                                                                        ]    9.60M    1.04MB/s
openFrameworksLibs_b  39% [=============================================>                                                                       ]    7.73M    1.19MB/s
openFrameworksLibs_b  15% [=================>                                                                                                   ]    8.33M    1.24MB/s
                          [Files: 0  Bytes: 0  [0 B/s] Redirects: 3  Todo: 3  Errors: 0                                                         ]

@danoli3
Copy link
Member Author

danoli3 commented May 27, 2024

Yeah!! 2>/dev/null was preventing the visualisation!

cURL's parallel needed some work though, fixed that up properly

@danoli3
Copy link
Member Author

danoli3 commented May 27, 2024

cURL

--Downloading using cURL urls [https://github.com/openframeworks/apothecary/releases/download/nightly/openFrameworksLibs_master_ios1.tar.bz2 -o openFrameworksLibs_master_ios1.tar.bz2 -k https://github.com/openframeworks/apothecary/releases/download/nightly/openFrameworksLibs_master_ios2.tar.bz2 -o openFrameworksLibs_master_ios2.tar.bz2 -k https://github.com/openframeworks/apothecary/releases/download/nightly/openFrameworksLibs_master_ios3.tar.bz2 -o openFrameworksLibs_master_ios3.tar.bz2 -k https://github.com/openframeworks/apothecary/releases/download/nightly/openFrameworksLibs_master_ios4.tar.bz2 -o openFrameworksLibs_master_ios4.tar.bz2 ]
DL% UL%  Dled  Uled  Xfers  Live Total     Current  Left    Speed

--  --      0     0     4     4  --:--:-- --:--:-- --:--:--     0      
--  --      0     0     4     4  --:--:-- --:--:-- --:--:--     0      
 30 --  36.9M     0     4     4   0:00:03  0:00:01  0:00:02 35.3M      
 53 --  65.4M     0     4     2   0:00:02  0:00:01  0:00:01 42.3M      
 76 --  93.3M     0     4     1   0:00:02  0:00:02 --:--:-- 45.5M      
 99 --   122M     0     4     1   0:00:02  0:00:02 --:--:-- 47.9M      
100 --   122M     0     4     0   0:00:02  0:00:02 --:--:-- 47.9M     

wget2
Screenshot 2024-05-28 at 7 55 04 am

@dimitre
Copy link
Member

dimitre commented May 27, 2024

woww that's great! I'm testing now in macOS and it is running great with wget2.
download_libs.sh 100%
and download_latest_libs.sh is downloading them twice.
here is the output

d@zen osx % ./download_latest_libs.sh
 -----
https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_1.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_2.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2
 [downloader] ... 
/opt/homebrew/bin/wget2
/opt/homebrew/bin/wget2
/opt/homebrew/bin/wget2

/opt/homebrew/bin/wget2
openFrameworksLibs_b 100% [====================================================================================================================>]   18.52M    1.20MB/s
openFrameworksLibs_b 100% [====================================================================================================================>]   24.50M    2.00MB/s
openFrameworksLibs_b 100% [====================================================================================================================>]    6.92M    1.20MB/s
                          [Files: 3  Bytes: 49.95M [2.66MB/s] Redirects: 3  Todo: 0  Errors: 0                                                  ]

 
 Removing old libraries
 ------ 
 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_1.tar.bz2]
 Deployed libraries from [openFrameworksLibs_bleeding_osx_1.tar.bz2] to [/libs]
 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_2.tar.bz2]
opencv/lib/osx/opencv.xcframework/macos-arm64_x86_64/libopencv_combined.a: truncated bzip2 input
tar: Error exit delayed from previous errors.
Error on or near line 262; exiting with status 1
../dev/download_libs.sh: line 70: [: too many arguments
 -----
https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_1.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_2.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2
 [downloader] ... 
/opt/homebrew/bin/wget2
/opt/homebrew/bin/wget2
/opt/homebrew/bin/wget2

/opt/homebrew/bin/wget2
openFrameworksLibs_b 100% [====================================================================================================================>]   18.52M    1.30MB/s
openFrameworksLibs_b 100% [====================================================================================================================>]   24.50M    1.87MB/s
openFrameworksLibs_b 100% [====================================================================================================================>]    6.92M    1.02MB/s
                          [Files: 3  Bytes: 49.95M [2.65MB/s] Redirects: 3  Todo: 0  Errors: 0                                                  ]

 ------ 
 Uncompressing libraries [osx]_[arm64] from [openFrameworksLibs_bleeding_osx_1.tar.bz2]
 Deployed libraries from [openFrameworksLibs_bleeding_osx_1.tar.bz2] to [/libs]
 Uncompressing libraries [osx]_[arm64] from [openFrameworksLibs_bleeding_osx_2.tar.bz2]
opencv/lib/osx/opencv.xcframework/macos-arm64_x86_64/libopencv_combined.a: truncated bzip2 input
tar: Error exit delayed from previous errors.
Error on or near line 262; exiting with status 1
../dev/download_libs.sh: line 70: [: too many arguments

@dimitre
Copy link
Member

dimitre commented May 28, 2024

Great. downloading once now. it seems to be downloading x86_64 here instead of arm
and some other errors uncompressing bzip

 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_1.tar.bz2]
 Deployed libraries from [openFrameworksLibs_bleeding_osx_1.tar.bz2] to [/libs]
 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_2.tar.bz2]
opencv/lib/osx/opencv.xcframework/macos-arm64_x86_64/libopencv_combined.a: truncated bzip2 input
tar: Error exit delayed from previous errors.
Error on or near line 262; exiting with status 1
../dev/download_libs.sh: line 70: [: too many arguments

@danoli3
Copy link
Member Author

danoli3 commented May 28, 2024

Been testing on a few computers.

  1. Yeah on Windows wget2 getting a strange timeout:
B%20filename%3DopenFrameworksLibs_bleeding_vs_64_2.zip&response-content-type=application%2Foctet-stream
[1] Downloading 'https://objects.githubusercontent.com/github-production-release-asset-2e65be/10855588/69d920c8-c16d-40d4-90fe-5a187476ed90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240528%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240528T002455Z&X-Amz-Expires=300&X-Amz-Signature=1184c22008fb3bff11bfec47782c5ff72e96be8448323bb695c80aa173041478&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=10855588&response-content-disposition=attachment%3B%20filename%3DopenFrameworksLibs_bleeding_vs_64_2.zip&response-content-type=application%2Foctet-stream' ...
[0] Downloading 'https://objects.githubusercontent.com/github-production-release-asset-2e65be/10855588/b9c15e1d-a4e2-41e3-8112-81b4118a52c3?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240528%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240528T002455Z&X-Amz-Expires=300&X-Amz-Signature=c3e2cec46b247b46dcc20b004dd032a7427a2008596391324f088210a1735f3c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=10855588&response-content-disposition=attachment%3B%20filename%3DopenFrameworksLibs_bleeding_vs_64_1.zip&response-content-type=application%2Foctet-stream' ...
Saving 'openFrameworksLibs_bleeding_vs_64_2.zip'
Saving 'openFrameworksLibs_bleeding_vs_64_1.zip'
HTTP response 200  [https://objects.githubusercontent.com/github-production-release-asset-2e65be/10855588/69d920c8-c16d-40d4-90fe-5a187476ed90?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240528%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240528T002455Z&X-Amz-Expires=300&X-Amz-Signature=1184c22008fb3bff11bfec47782c5ff72e96be8448323bb695c80aa173041478&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=10855588&response-content-disposition=attachment%3B%20filename%3DopenFrameworksLibs_bleeding_vs_64_2.zip&response-content-type=application%2Foctet-stream]
Error on or near line ; exiting with status 1
../dev/download_libs.sh: line 70: [: too many arguments

I was watching the cache files as well, it was downloading them and got stuck and some sort of a loop and stopped getting my bytes, looks like remote hung up on the transfer. Prior calls were fine with wget2, even to the same binaries

  1. Also need to fix these random logs:
/opt/homebrew/bin/wget2

Side effect of checking if command function exists >= null should have handled that

  1. Also if failure the in progress files go into the script dev directory... we should allocate a scrub folder

**
image
**
I think once it starts getting .1 our code to extract them is broke

- fixes command determination artefact now tracks installed of curl/wget2/wget in variable
- added wget2 params for override or continue last download (-nc)
- t20 retries
- console formatting
- added error message if no commands installed
@danoli3
Copy link
Member Author

danoli3 commented May 28, 2024

Downloader updates

  • fixes command determination artefact now tracks installed of curl/wget2/wget in variable
  • added wget2 params for override or continue last download (-nc)
  • t20 retries
  • console formatting
  • added error message if no commands installed

@dimitre
Copy link
Member

dimitre commented May 28, 2024

have you seen this one? https://amber-lang.com

@dimitre
Copy link
Member

dimitre commented May 28, 2024

I've just tested it seems to be working great here.
the script now is showing like it is downloading x86_64 libs, but I'm on arm64

 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_1.tar.bz2]
 Deployed libraries from [openFrameworksLibs_bleeding_osx_1.tar.bz2] to [/libs]
 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_2.tar.bz2]
 Deployed libraries from [openFrameworksLibs_bleeding_osx_2.tar.bz2] to [/libs]
 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_3.tar.bz2]
 Deployed libraries from [openFrameworksLibs_bleeding_osx_3.tar.bz2] to [/libs]

a nice glob command to test all libs architectures

cd $ofw/libs
file **/*.a

and the results

FreeImage/lib/osx/FreeImage.xcframework/macos-arm64_x86_64/FreeImage.a: Mach-O universal binary with 2 architectures: [x86_64:current ar archive] [arm64]
FreeImage/lib/osx/FreeImage.xcframework/macos-arm64_x86_64/FreeImage.a (for architecture x86_64):	current ar archive
FreeImage/lib/osx/FreeImage.xcframework/macos-arm64_x86_64/FreeImage.a (for architecture arm64):	current ar archive
brotli/lib/osx/brotli.xcframework/macos-arm64_x86_64/brotli.a:          Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
brotli/lib/osx/brotli.xcframework/macos-arm64_x86_64/brotli.a (for architecture x86_64):	current ar archive random library
brotli/lib/osx/brotli.xcframework/macos-arm64_x86_64/brotli.a (for architecture arm64):	current ar archive random library
cairo/lib/osx/cairo.xcframework/macos-arm64_x86_64/libcairo.a:          Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64]
cairo/lib/osx/cairo.xcframework/macos-arm64_x86_64/libcairo.a (for architecture x86_64):	current ar archive random library
cairo/lib/osx/cairo.xcframework/macos-arm64_x86_64/libcairo.a (for architecture arm64):	current ar archive random library
curl/lib/osx/curl.xcframework/macos-arm64_x86_64/curl.a:                Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
curl/lib/osx/curl.xcframework/macos-arm64_x86_64/curl.a (for architecture x86_64):	current ar archive random library
curl/lib/osx/curl.xcframework/macos-arm64_x86_64/curl.a (for architecture arm64):	current ar archive random library
fmt/lib/osx/fmt.xcframework/macos-arm64_x86_64/libfmt.a:                Mach-O universal binary with 2 architectures: [x86_64:current ar archive] [arm64:current ar archive]
fmt/lib/osx/fmt.xcframework/macos-arm64_x86_64/libfmt.a (for architecture x86_64):	current ar archive
fmt/lib/osx/fmt.xcframework/macos-arm64_x86_64/libfmt.a (for architecture arm64):	current ar archive
freetype/lib/osx/freetype.xcframework/macos-arm64_x86_64/libfreetype.a: Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
freetype/lib/osx/freetype.xcframework/macos-arm64_x86_64/libfreetype.a (for architecture x86_64):	current ar archive random library
freetype/lib/osx/freetype.xcframework/macos-arm64_x86_64/libfreetype.a (for architecture arm64):	current ar archive random library
glew/lib/osx/glew.xcframework/macos-arm64_x86_64/libGLEW.a:             Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64]
glew/lib/osx/glew.xcframework/macos-arm64_x86_64/libGLEW.a (for architecture x86_64):	current ar archive random library
glew/lib/osx/glew.xcframework/macos-arm64_x86_64/libGLEW.a (for architecture arm64):	current ar archive random library
glfw/lib/osx/glfw.xcframework/macos-arm64_x86_64/libglfw3.a:            Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
glfw/lib/osx/glfw.xcframework/macos-arm64_x86_64/libglfw3.a (for architecture x86_64):	current ar archive random library
glfw/lib/osx/glfw.xcframework/macos-arm64_x86_64/libglfw3.a (for architecture arm64):	current ar archive random library
libpng/lib/osx/libpng.xcframework/macos-arm64_x86_64/libpng.a:          Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
libpng/lib/osx/libpng.xcframework/macos-arm64_x86_64/libpng.a (for architecture x86_64):	current ar archive random library
libpng/lib/osx/libpng.xcframework/macos-arm64_x86_64/libpng.a (for architecture arm64):	current ar archive random library
openFrameworksCompiled/lib/osx/libopenFrameworks.a:                     current ar archive
pixman/lib/osx/pixman.xcframework/macos-arm64_x86_64/libpixman-1.a:     Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
pixman/lib/osx/pixman.xcframework/macos-arm64_x86_64/libpixman-1.a (for architecture x86_64):	current ar archive random library
pixman/lib/osx/pixman.xcframework/macos-arm64_x86_64/libpixman-1.a (for architecture arm64):	current ar archive random library
pugixml/lib/osx/pugixml.xcframework/macos-arm64_x86_64/libpugixml.a:    Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
pugixml/lib/osx/pugixml.xcframework/macos-arm64_x86_64/libpugixml.a (for architecture x86_64):	current ar archive random library
pugixml/lib/osx/pugixml.xcframework/macos-arm64_x86_64/libpugixml.a (for architecture arm64):	current ar archive random library
rtAudio/lib/osx/rtAudio.xcframework/macos-arm64_x86_64/librtaudio.a:    Mach-O universal binary with 2 architectures: [x86_64:current ar archive] [arm64:current ar archive]
rtAudio/lib/osx/rtAudio.xcframework/macos-arm64_x86_64/librtaudio.a (for architecture x86_64):	current ar archive
rtAudio/lib/osx/rtAudio.xcframework/macos-arm64_x86_64/librtaudio.a (for architecture arm64):	current ar archive
tess2/lib/osx/tess2.xcframework/macos-arm64_x86_64/libtess2.a:          Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
tess2/lib/osx/tess2.xcframework/macos-arm64_x86_64/libtess2.a (for architecture x86_64):	current ar archive random library
tess2/lib/osx/tess2.xcframework/macos-arm64_x86_64/libtess2.a (for architecture arm64):	current ar archive random library
uriparser/lib/osx/uriparser.xcframework/macos-arm64_x86_64/uriparser.a: Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
uriparser/lib/osx/uriparser.xcframework/macos-arm64_x86_64/uriparser.a (for architecture x86_64):	current ar archive random library
uriparser/lib/osx/uriparser.xcframework/macos-arm64_x86_64/uriparser.a (for architecture arm64):	current ar archive random library
zlib/lib/osx/zlib.xcframework/macos-arm64_x86_64/zlib.a:                Mach-O universal binary with 2 architectures: [x86_64:current ar archive random library] [arm64:current ar archive random library]
zlib/lib/osx/zlib.xcframework/macos-arm64_x86_64/zlib.a (for architecture x86_64):	current ar archive random library
zlib/lib/osx/zlib.xcframework/macos-arm64_x86_64/zlib.a (for architecture arm64):	current ar archive random library

@dimitre
Copy link
Member

dimitre commented May 28, 2024

it would be great if we had ./scripts/osx/download_libs.sh using new downloader also :)

@danoli3
Copy link
Member Author

danoli3 commented May 28, 2024

You should run file **/*.a on the ultra mega xcframework with the macos scripts, its massif.

it would be great if we had ./scripts/osx/download_libs.sh using new downloader also :)

Its already powering that too

Fixed: [osx]_[x86_64]

@dimitre
Copy link
Member

dimitre commented May 28, 2024

I had this error now:

./scripts/osx/download_latest_libs.sh
 -----
https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_1.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_2.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2
 [openFrameworks downloader] ... 

[0] Downloading 'https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2' ...
HTTP response 302  [https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2]
Adding URL: https://objects.githubusercontent.com/github-production-release-asset-2e65be/10855588/2b3f3f8d-1563-498a-aa13-9da7293b2d56?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240528%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240528T195402Z&X-Amz-Expires=300&X-Amz-Signature=66771b667640bbfde32eea8c9c2a0dc3d1abddca2ccdd27eef6caa843eeadda9&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=10855588&response-content-disposition=attachment%3B%20filename%3DopenFrameworksLibs_bleeding_osx_3.tar.bz2&response-content-type=application%2Foctet-stream
[0] Downloading 'https://objects.githubusercontent.com/github-production-release-asset-2e65be/10855588/2b3f3f8d-1563-498a-aa13-9da7293b2d56?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240528%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240528T195402Z&X-Amz-Expires=300&X-Amz-Signature=66771b667640bbfde32eea8c9c2a0dc3d1abddca2ccdd27eef6caa843eeadda9&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=10855588&response-content-disposition=attachment%3B%20filename%3DopenFrameworksLibs_bleeding_osx_3.tar.bz2&response-content-type=application%2Foctet-stream' ...
Saving 'openFrameworksLibs_bleeding_osx_3.tar.bz2'
HTTP response 200  [https://objects.githubusercontent.com/github-production-release-asset-2e65be/10855588/2b3f3f8d-1563-498a-aa13-9da7293b2d56?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240528%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240528T195402Z&X-Amz-Expires=300&X-Amz-Signature=66771b667640bbfde32eea8c9c2a0dc3d1abddca2ccdd27eef6caa843eeadda9&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=10855588&response-content-disposition=attachment%3B%20filename%3DopenFrameworksLibs_bleeding_osx_3.tar.bz2&response-content-type=application%2Foctet-stream]

 
 Removing old libraries
 ------ 
 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_1.tar.bz2]
FreeImage/lib/osx/FreeImage.xcframework/macos-arm64_x86_64/FreeImage.a: truncated bzip2 input
tar: Error exit delayed from previous errors.
Error on or near line 262; exiting with status 1

after removing all libs folder it stopped working.
download_libs.sh worked ok

and subsequent calling download_latest_libs.sh returns

./scripts/osx/download_latest_libs.sh
 -----
https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_1.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_2.tar.bz2 https://github.com/openframeworks/apothecary/releases/download/bleeding/openFrameworksLibs_bleeding_osx_3.tar.bz2
 [openFrameworks downloader] ... 

Nothing to do - goodbye

 
 Removing old libraries
 ------ 
 Uncompressing libraries [osx]_[x86_64] from [openFrameworksLibs_bleeding_osx_1.tar.bz2]
FreeImage/lib/osx/FreeImage.xcframework/macos-arm64_x86_64/FreeImage.a: truncated bzip2 input
tar: Error exit delayed from previous errors.
Error on or near line 262; exiting with status 1

@dimitre
Copy link
Member

dimitre commented May 29, 2024

Good idea! version is great

[openFrameworks downloader v3.0.1] ... 


Nothing to do - goodbye

 
 Overwrite - Removing old libraries for [osx]
 Removing old libraries: [boost/lib/osx]
 Removing old libraries: [cairo/lib/osx]
 Removing old libraries: [curl/lib/osx]
 Removing old libraries: [FreeImage/lib/osx]
 Removing old libraries: [fmod/lib/osx]
 Removing old libraries: [freetype/lib/osx]
 Removing old libraries: [glew/lib/osx]
 Removing old libraries: [glfw/lib/osx]
 Removing old libraries: [rtAudio/lib/osx]
 Removing old libraries: [tess2/lib/osx]
 Removing old libraries: [uriparser/lib/osx]
 ------ 
 Uncompressing libraries [osx] from [openFrameworksLibs_bleeding_osx_1.tar.bz2]
FreeImage/lib/osx/FreeImage.xcframework/macos-arm64_x86_64/FreeImage.a: truncated bzip2 input
tar: Error exit delayed from previous errors.
Error on or near line 263; exiting with status 1
d@zen of % ./scripts/osx/download_latest_libs.sh
 openFrameworks download_libs.sh v2.0
 -----
 [openFrameworks downloader v3.0.1] ... 

@danoli3
Copy link
Member Author

danoli3 commented May 29, 2024

Nothing to do - goodbye
this means theres a local file downloaded already and its corrupted, hmmmm okay another check

@dimitre
Copy link
Member

dimitre commented May 29, 2024

Yes maybe only because I've tested scripts under constructions, but it would be great to handle those also, in the case it happens somewhere else.
Screenshot 2024-05-29 at 14 30 19

- files downloaded (zip/tar.bz2) are now stored in libs/download/
- creates libs/download for downloading zips/tars for optimisation
- clean up of logs
- not removing dl zip now
- code to remove zips on start also done commented out
@danoli3
Copy link
Member Author

danoli3 commented May 29, 2024

I thought I'd add another folder to libs

Screenshot 2024-05-30 at 4 47 13 am

downloader 3.1.0 wget/wget2 - now checks remote file vs local if found and optimises - won't download if same

download_libs 2.1 -

  • files downloaded (zip/tar.bz2) are now stored in openFrameworks/libs/download/
  • not removing dl (zip/tar.bz2) now after extraction
  • code to remove (zip/tar.bz2) on start also done - disabled

@danoli3
Copy link
Member Author

danoli3 commented May 29, 2024

Screenshot 2024-05-29 at 10 26 47 pm

cURLs progress bar

@dimitre
Copy link
Member

dimitre commented May 29, 2024

Beautiful. install complete here. didnt have to remove any file.
Installed latest libs, worked great
installed classic libs, all good
Terminal output is pretty!

All very fast here in a slow rural internet connection via radio.

Suggestions:
I would love to have a different starting name for both scripts, to complete with tab, like "latest_libs.sh" or something

Thanks for all the work. it is great here in macOS. I can test later in Windows

@dimitre
Copy link
Member

dimitre commented May 29, 2024

Good to go. only reminding you of file permissions
#7907 (comment)

@danoli3
Copy link
Member Author

danoli3 commented May 30, 2024

Linux 24.04 issues on restart actions/runner-images#9937

@danoli3 danoli3 merged commit 676ae97 into openframeworks:master May 30, 2024
11 checks passed
@danoli3
Copy link
Member Author

danoli3 commented May 30, 2024

Sweet merging as fixes linux actions as well

@dimitre
Copy link
Member

dimitre commented May 30, 2024

Yes! I've fixed it by removing sudo apt update which calls some services to restart, including the github actions.
I'll do the same for projectGenerator

danoli3 added a commit to danoli3/openFrameworks that referenced this pull request Jun 6, 2024
…frameworks

* commit '25d0a4ba0c8b8ee96123fc669a884ab9f3f77982':
  Download ProjectGenerator to downloads, optimise wont re-download if in sync (openframeworks#8012)
  ofxAssimpModelLoader addon_config.mk osx exclusions
  Downloader 3.2.3 - fixes for install detection and single url for curl (openframeworks#8009)
  Download libs script update (adds openframeworks#7907 wget2) - params and fixes deploy logs  (openframeworks#8002)
  .mk removing repeated configs
  COREVIDEO_SILENCE_GL_DEPRECATION (openframeworks#8005)
  ios template update
  update addons folder in xcode template (openframeworks#8003)
  VS download latest x64 only (openframeworks#7999)

# Conflicts:
#	scripts/apothecary
#	scripts/dev/download_libs.sh
#	scripts/osx/download_latest_libs.sh
#	scripts/templates/ios/emptyExample.xcodeproj/project.pbxproj
danoli3 added a commit to danoli3/openFrameworks that referenced this pull request Jun 18, 2024
* master: (244 commits)
  ofxAssimpModelLoader addon_config.mk osx exclusions
  Downloader 3.2.3 - fixes for install detection and single url for curl (openframeworks#8009)
  Downloader 3.2.3 - fixes for install detection and single url for curl
  Download libs script update (adds openframeworks#7907 wget2) - params and fixes deploy logs  (openframeworks#8002)
  .mk removing repeated configs
  COREVIDEO_SILENCE_GL_DEPRECATION (openframeworks#8005)
  ios template update
  update addons folder in xcode template (openframeworks#8003)
  VS download latest x64 only (openframeworks#7999)
  Android 2024 template (openframeworks#7991)
  XCode macOS template update
  up (openframeworks#7996)
  msys2 improvements (openframeworks#7976)
  commented out .h files removed (openframeworks#7990)
  VS2022 FreeType Fix (openframeworks#7992)
  Update ofPixels.cpp (openframeworks#7989)
  ofAppNoWindow..h: #include ofEvents to complete the ofCoreEvents type (openframeworks#7943)
  ofFile::doesFileExist simplified (openframeworks#7970)
  fix unneeded reference for `add` of `bool`, `char` (openframeworks#7973)
  ofxOscReceiver: win32 implementation: shadow posix changes (openframeworks#7979)
  ...

# Conflicts:
#	.gitignore
#	libs/openFrameworks/gl/ofGLProgrammableRenderer.cpp
#	libs/openFrameworks/gl/ofShader.cpp
#	libs/openFrameworks/gl/ofTexture.h
#	libs/openFrameworks/graphics/ofGraphicsBaseTypes.cpp
#	libs/openFrameworks/graphics/ofTrueTypeFont.cpp
#	libs/openFrameworks/graphics/ofTrueTypeFont.h
#	libs/openFrameworks/sound/ofAVEngineSoundPlayer.h
#	libs/openFrameworks/sound/ofSoundBaseTypes.h
#	libs/openFrameworks/utils/ofConstants.h
#	libs/openFrameworks/utils/ofThread.h
#	libs/openFrameworks/utils/ofURLFileLoader.cpp
#	libs/openFrameworks/utils/ofUtils.h
#	libs/openFrameworks/utils/ofXml.h
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants