py-tensorflow: Update to version 2.12.0, Add Python 310 #15397

essandess · 2022-07-18T10:29:00Z

Description

Depends upon bazel: Bugfix, Update to version 6.1.2; PG bazel #16149.

Type(s)

bugfix
enhancement
security fix

Tested on

macOS 12.4 21F79 x86_64
Xcode 13.4.1 13F100

Verification

Have you

followed our Commit Message Guidelines?
squashed and minimized your commits?
checked that there aren't other open pull requests for the same change?
referenced existing tickets on Trac with full URL?
checked your Portfile with port lint --nitpick?
tried existing tests with sudo port test?
tried a full install with sudo port -vst install?
tested basic functionality of all binary files?

macportsbot · 2022-07-18T10:29:04Z

Notifying maintainers:
@cjones051073 for port bazel, py-tensorflow.
@missa-prime for port bazel.
@emcrisostomo for port py-tensorflow.

devel/bazel/Portfile

mascguy · 2022-07-20T13:36:41Z

Steve, I ran a quick local test on 10.15, to help validate whether the CI failures are legitimate. And sure enough, bazel is failing to build for me as well.

essandess · 2022-07-20T14:57:29Z

Steve, I ran a quick local test on 10.15, to help validate whether the CI failures are legitimate. And sure enough, bazel is failing to build for me as well.

I only have access to macOS 12, and the build works. I expect that's an upstream issue that we'll never have much control over, or cycles to fix (per https://lists.macports.org/pipermail/macports-dev/2022-July/044443.html).

I suggest just getting this thing running on the latest macOS and call it a day. If anyone requires bazel on older macOS version and knows how to fix it, or wants to download the binaries directly, they can issue a PR for an updated approach.

essandess · 2022-07-30T15:50:58Z

Upstream offers three options to address old bazel version issues:

Figure out a patch ourselves
Use a tool called bazelisk to download and install previous binaries
Get any repos that use prior bazel versions to support bazel 5

None of these are viable options at least with the resources I have. This supports adoption of the strategy laid out above:

just get this thing running on the latest macOS and call it a day. If anyone requires bazel on older macOS version and knows how to fix it, or wants to download the binaries directly, they can issue a PR for an updated approach.

mascguy · 2022-09-02T12:44:37Z

Rebased against master.

Also testing locally on 10.15, Big Sur, and Monterey, to see what the results are. More to follow.

python/py-tensorflow/Portfile

essandess · 2022-09-17T20:54:04Z

This now fails locally with the linker error:

:info:build # Execution platform: @local_execution_config_platform//:platform
:info:build ld: malformed trie, childNodeOffset==0 file 'bazel-out/darwin-opt/bin/_solib_darwin_x86_64/_U_S_Stensorflow_Spython_C_Upywrap_Utensorflow_Uinternal_Umacos___Utensorflow_Spython/lib_pywrap_tensorflow_internal.dylib'
:info:build clang: error: linker command failed with exit code 1 (use -v to see invocation)
:info:build Error in child process '/usr/bin/xcrun'. 1

I'm throwing in the towel getting tensorflow built, and will split off the bazel commits, which do build and run.

reneeotten · 2023-05-07T13:52:52Z

I'm fine with removing the older Python versions, but then add at least Python 3.11 which is MacPort's default

mascguy · 2023-05-07T14:02:36Z

I'm fine with removing the older Python versions, but then add at least Python 3.11 which is MacPort's default

That's going to require adding Python 3.11 support to a number of dependencies, which don't yet exist. So I don't think that's feasible for this update:

$ port lint --nitpick py311-tensorflow
--->  Verifying Portfile for py311-tensorflow
Error: Unknown dependency: py311-absl
Error: Unknown dependency: py311-astunparse
Error: Unknown dependency: py311-google-pasta
Error: Unknown dependency: py311-grpcio
Error: Unknown dependency: py311-keras
Error: Unknown dependency: py311-opt_einsum
Error: Unknown dependency: py311-protobuf3
Error: Unknown dependency: py311-tensorboard
Error: Unknown dependency: py311-tensorflow_estimator

…onflict declarations

mascguy · 2023-05-07T14:07:47Z

Okay, I'm finished with my cleanup, and I think we've done as much pre-testing as we can. Let's merge, and let the buildbots take on this update.

See: #15397

mascguy · 2023-05-07T14:32:46Z

The builds are failing very early in the process, for Monterey and Ventura Intel. (Still waiting to see whether it fails for Big Sur, as well as for ARM.)

Though the errors are slightly different between them:

mascguy · 2023-05-07T14:38:48Z

Crap, it's also failing for ARM, albeit during the dependency installation phase. Hopefully this is easy to fix, as I thought (?) we had ARM support for some of our Java ports:

----> Installing dependency (17 of 78) 'bazel' with variants ''
--->  Fetching distfiles for bazel
DEBUG: java-portgroup: Trying to find JVM version: 11
DEBUG: No value for java JAVA_HOME was automatically discovered
DEBUG: Adding dependency on JDK fallback openjdk11-zulu
Error: bazel requires Java 11 but no such installation could be found.
Error: Failed to fetch bazel: missing required Java version

https://build.macports.org/builders/ports-13_arm64-builder/builds/21046/steps/install-dependencies/logs/stdio

catap · 2023-05-07T14:40:27Z

So, someone had broke java for arm?

mascguy · 2023-05-07T14:41:51Z

So, someone had broke java for arm?

Hopefully not, this might simply be a bug/oversight in the Java portgroup. Need to investigate further.

catap · 2023-05-07T14:51:11Z

catap@Kirills-mini-m1 ~ % sudo port install openjdk11-zulu
--->  Fetching archive for openjdk11-zulu
--->  Attempting to fetch openjdk11-zulu-11.64.19_0.darwin_22.arm64.tbz2 from https://packages.macports.org/openjdk11-zulu
--->  Attempting to fetch openjdk11-zulu-11.64.19_0.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/openjdk11-zulu
--->  Attempting to fetch openjdk11-zulu-11.64.19_0.darwin_22.arm64.tbz2 from https://fra.de.packages.macports.org/openjdk11-zulu
--->  Fetching distfiles for openjdk11-zulu
--->  Attempting to fetch zulu11.64.19-ca-jdk11.0.19-macosx_aarch64.tar.gz from https://cdn.azul.com/zulu/bin/
--->  Verifying checksums for openjdk11-zulu                                         
--->  Extracting openjdk11-zulu
--->  Configuring openjdk11-zulu
--->  Building openjdk11-zulu
--->  Staging openjdk11-zulu into destroot
Warning: openjdk11-zulu installs files outside the common directory structure.
--->  Installing openjdk11-zulu @11.64.19_0

Seems that openjdk11-zulu wasn't build by Buildbot...

mascguy · 2023-05-07T14:53:28Z

The builds are failing very early in the process, for Monterey and Ventura Intel. (Still waiting to see whether it fails for Big Sur, as well as for ARM.)

Details for Monterey (paths trimmed to be relative, for readability):

Repository rule llvm_configure defined at:
  py310-tensorflow/work/bazel_build/149bd4da7eceb9ef1c48eadfd6b093af/external/llvm-raw/utils/bazel/configure.bzl:169:33: in <toplevel>
Analyzing: target //tensorflow/tools/pip_package:build_pip_package (53 packages loaded, 12 targets configured)
ERROR: An error occurred during the fetch of repository 'llvm-project':
   Traceback (most recent call last):
	File "py310-tensorflow/work/bazel_build/149bd4da7eceb9ef1c48eadfd6b093af/external/llvm-raw/utils/bazel/configure.bzl", line 146, column 25, in _llvm_configure_impl
		_overlay_directories(repository_ctx)
	File "py310-tensorflow/work/bazel_build/149bd4da7eceb9ef1c48eadfd6b093af/external/llvm-raw/utils/bazel/configure.bzl", line 64, column 13, in _overlay_directories
		fail(("Failed to execute overlay script: '{cmd}'\n" +
Error in fail: Failed to execute overlay script:
'/usr/bin/python3 py310-tensorflow/work/bazel_build/149bd4da7eceb9ef1c48eadfd6b093af/external/llvm-raw/utils/bazel/overlay_directories.py --src py310-tensorflow/work/bazel_build/149bd4da7eceb9ef1c48eadfd6b093af/external/llvm-raw --overlay py310-tensorflow/work/bazel_build/149bd4da7eceb9ef1c48eadfd6b093af/external/llvm-raw/utils/bazel/llvm-project-overlay --target .'

Details for Ventura (again, paths trimmed to relative):

Loading: 96 packages loaded
    currently loading: tensorflow ... (26 packages)
WARNING: Download from https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/XNNPACK/archive/659147817805d17c7be2d60bd7bbca7e780f9c82.zip failed:
    class java.io.FileNotFoundException GET returned 404 Not Found

INFO: Repository llvm-project instantiated at:
  py310-tensorflow/work/tensorflow-2.12.0/WORKSPACE:15:14: in <toplevel>
  py310-tensorflow/work/tensorflow-2.12.0/tensorflow/workspace2.bzl:967:21: in workspace
  py310-tensorflow/work/tensorflow-2.12.0/tensorflow/workspace2.bzl:539:15: in _tf_repositories
  py310-tensorflow/work/tensorflow-2.12.0/third_party/llvm/setup.bzl:22:19: in llvm_setup
Repository rule llvm_configure defined at:
  py310-tensorflow/work/bazel_build/149bd4da7eceb9ef1c48eadfd6b093af/external/llvm-raw/utils/bazel/configure.bzl:169:33: in <toplevel>
Loading: 99 packages loaded
    currently loading: tensorflow ... (27 packages)

mascguy · 2023-05-07T14:58:09Z

The builds are failing very early in the process, for Monterey and Ventura Intel. (Still waiting to see whether it fails for Big Sur, as well as for ARM.)

One bit of good news: So far, the Big Sur build is proceeding, without the failures seen for Monterey and Ventura. Hopefully it will succeed, keep your fingers crossed...

mascguy · 2023-05-07T15:17:19Z

catap@Kirills-mini-m1 ~ % sudo port install openjdk11-zulu
--->  Fetching archive for openjdk11-zulu
--->  Attempting to fetch openjdk11-zulu-11.64.19_0.darwin_22.arm64.tbz2 from https://packages.macports.org/openjdk11-zulu
--->  Attempting to fetch openjdk11-zulu-11.64.19_0.darwin_22.arm64.tbz2 from https://nue.de.packages.macports.org/openjdk11-zulu
--->  Attempting to fetch openjdk11-zulu-11.64.19_0.darwin_22.arm64.tbz2 from https://fra.de.packages.macports.org/openjdk11-zulu
--->  Fetching distfiles for openjdk11-zulu
--->  Attempting to fetch zulu11.64.19-ca-jdk11.0.19-macosx_aarch64.tar.gz from https://cdn.azul.com/zulu/bin/
--->  Verifying checksums for openjdk11-zulu                                         
--->  Extracting openjdk11-zulu
--->  Configuring openjdk11-zulu
--->  Building openjdk11-zulu
--->  Staging openjdk11-zulu into destroot
Warning: openjdk11-zulu installs files outside the common directory structure.
--->  Installing openjdk11-zulu @11.64.19_0

Seems that openjdk11-zulu wasn't build by Buildbot...

I think that was intentional, though perhaps not ideal. Theoretically that shouldn't cause the issue though; I'm guessing the Java PG isn't finding the installation.

mascguy · 2023-05-07T15:25:34Z

I think that was intentional, though perhaps not ideal. Theoretically that shouldn't cause the issue though; I'm guessing the Java PG isn't finding the installation.

On further review, it looks like the Java PG is working correctly, and pulling in openjdk11-zulu for ARM.

Instead, it appears that Bazel (the project, not our port logic) isn't finding the JDK. Still digging...

catap · 2023-05-07T18:00:22Z

BTW I quite supprised that bazel doesn't build JDK from scratch. Seem that it something which they need to work on the next release :)

mascguy · 2023-05-07T19:34:23Z

The builds are failing very early in the process, for Monterey and Ventura Intel. (Still waiting to see whether it fails for Big Sur, as well as for ARM.)

Hmmmm... well, on the 2nd time around, the build for Monterey Intel didn't fail like it did before. So perhaps some upstream server was down, when files were being pulled? (Something in the realm of https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/....)

So there's hope that the Ventura Intel build might do the same. Fingers crossed...

essandess · 2023-05-07T19:36:05Z

That’s pretty impressive you all got this and the bazel port working. Thank you!

mascguy · 2023-05-08T17:31:01Z

Update: The builds for Bug Sur and Monterey almost complete successfully, but they fail at the very end with:

Execution platform: locul execution_config platform//:platform

2023-05-08 09:51:12.901685: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available cpu instructions in performance-critical of To enable the following instructions: SSE3 55E4.1 55E4.2, In other operations, rebuild tensorflow with the appropriate compiler flags.

Traceback (most recent call last):

File "py39 tensorflow/work/bazel build/e65bb5c129a76359f8df5ada6c4c0686/exerroot/org tensorflow/bazel-out/darwin-opt-exec-504E0418/
bin/tensorflow/create_tensorflow.python_api_t python_api_gen_v2.runfiles/org tensorflow/tensorflow/python/tools/api/generator/create_python api.py", line 22, in <module> From tensorflow.python.tools.api.generator import doc sres

File "py39-tensorflow/work/bazel build/e65bb5c129a76359f8d+Sade6c4c0686/execroot/org tensorflow/bazel-out/darwin opt exec-S04E0418/
bin/tensorflow/create tensorflow.python api tf python_api_gen_v2.runfiles/org tensorflow/tensorflow/python/init.py", line 37, in <module>

File "py39-tensorflow/work/bazel build/e65bb5c129a76359f8df5da6c4c0686/execroot/org/tensorflow/bazel-out/darwin-opt-exer-504F0418/
from tensorflow.python.eager Import context bin/tensorflow/create_tensorflow.python_api_t python_api_gen_v2.runfiles/org tensorflow/tensorflow/python/eager/context.py", line 20, In modules
from tensorflow.core.framework Import function_pb2

File "py39-tensorflow/work/bazel_build/e65bb5c129a76359fdf5ada6c4c0686/execroot/org tensorflow/bazel-out/daruin-opt-exec-504F0418/ bin/tensorflow/create_tensorflow.python_api tf python_api_gen_v2.runfiles/org tensorflow/tensorflow/core/framework/function_pb2.py", line 5, in codule> from google.protobuf. internal Import builder as builder

ImportError: cannot import name "builder from 'google.protobut.internal

(/opt/local/Library/Frameworks/Python, framework/Versions/3.9/lib/python3.5/site-packages/google/protobuf/internal/__init__.py)

Target //tensorflow/tools/pip_package:build_pip package failed to build

ERROR: py39-tensorflow/work/tensorflow-2.12.0/tensorflow/python/tools/BUILD:314:10 Middleman middlemen/tensorflow Spython Stools Ssaved Umodel ucl runfiles failed: (Exit 1): bash

catap · 2023-05-08T17:39:34Z

@mascguy I found the way to overstep that issue and it seems like insanity: https://stackoverflow.com/a/72494013

mascguy · 2023-05-08T22:04:28Z

@mascguy I found the way to overstep that issue and it seems like insanity: https://stackoverflow.com/a/72494013

Yeah, I found that article too. Very interesting!

During your research, did you see any definitive info, re: Which version of protobuf is required now? I ask, as upstream is in the 4.x territory, but we don't support that yet...

cjones051073 · 2023-05-09T20:55:30Z

@mascguy I found the way to overstep that issue and it seems like insanity: https://stackoverflow.com/a/72494013

Insanity is part of the course as far as I am concerned when it comes to Bazel based builds...

mascguy · 2023-05-09T20:59:36Z

@mascguy I found the way to overstep that issue and it seems like insanity: https://stackoverflow.com/a/72494013

Insanity is part of the course as far as I am concerned when it comes to Bazel based builds...

After taking an initial look at the documentation, and getting a better understanding of what it provides, etc, it seems like an incredibly powerful build tool! But with the caveat that the amount of functionality also requires a lot of learning.

I'm not saying it's perfect by any means - nor am I advocating that we expand its use - but it does bring a lot to the table...

catap · 2023-05-09T22:04:41Z

After taking an initial look at the documentation, and getting a better understanding of what it provides, etc, it seems like an incredibly powerful build tool! But with the caveat that the amount of functionality also requires a lot of learning.

I think that it is a kind of mistake to see bazel like a build tools. It is more like Docker or Nix which builds an application from different pieces.

It can be used to build an application, but it is more general things.

I'm not saying it's perfect by any means - nor am I advocating that we expand its use - but it does bring a lot to the table...

The issue that it brings the table with it. The specific table which has more than one a kind of vendor lock-in to google.

macportsbot assigned cjones051073 and emcrisostomo Jul 18, 2022

macportsbot added the maintainer: open Affects an openmaintainer port label Jul 18, 2022

essandess force-pushed the py-tensorflow branch 2 times, most recently from 2ca3f5f to c52e1e0 Compare July 18, 2022 13:51

mascguy self-assigned this Jul 19, 2022

mascguy reviewed Jul 19, 2022

View reviewed changes

devel/bazel/Portfile Outdated Show resolved Hide resolved

mascguy added the wip Work in progress label Jul 20, 2022

mascguy marked this pull request as draft July 20, 2022 13:36

essandess force-pushed the py-tensorflow branch from c52e1e0 to cff82ec Compare July 20, 2022 14:52

essandess mentioned this pull request Jul 22, 2022

py-transformers: Update to version 4.20.1, Add Python 310, Update dependencies #15456

Merged

11 tasks

mascguy force-pushed the py-tensorflow branch from cff82ec to c7a0975 Compare September 2, 2022 12:43

mascguy mentioned this pull request Sep 2, 2022

apache-arrow: Remove unnecessary libunwind dependency #15891

Merged

11 tasks

mascguy reviewed Sep 2, 2022

View reviewed changes

python/py-tensorflow/Portfile Outdated Show resolved Hide resolved

mascguy force-pushed the py-tensorflow branch 4 times, most recently from b3c71b5 to 0cef56c Compare September 2, 2022 20:13

essandess mentioned this pull request Sep 9, 2022

py-sentence-transformers: Update to version 2.2.2; Add Python 310, 311 #15601

Closed

11 tasks

mascguy force-pushed the py-tensorflow branch from 0cef56c to fead933 Compare September 10, 2022 13:15

essandess force-pushed the py-tensorflow branch from fead933 to 62369ed Compare September 17, 2022 18:04

essandess mentioned this pull request Sep 17, 2022

py-tensorflow-metal: Update to version 0.6.0; py-tensorflow-macos: Update to version 2.10.0 #16147

Merged

11 tasks

essandess mentioned this pull request Sep 17, 2022

bazel: Bugfix, Update to version 6.1.2; PG bazel #16149

Merged

11 tasks

essandess force-pushed the py-tensorflow branch from 62369ed to 32435d1 Compare September 17, 2022 20:58

py-tensorflow: fixes for build issues and speed

6d01d58

mascguy force-pushed the py-tensorflow branch from d4ec42d to 0c5dc00 Compare May 7, 2023 13:46

mascguy force-pushed the py-tensorflow branch from 0c5dc00 to d0fbbdb Compare May 7, 2023 13:56

py-tensorflow: drop python 3.7/3.8; allow xcode/clang 12.0; cleanup c…

9f06df6

…onflict declarations

mascguy force-pushed the py-tensorflow branch from d0fbbdb to 9f06df6 Compare May 7, 2023 14:04

mascguy merged commit fb29cdd into macports:master May 7, 2023

mascguy added a commit that referenced this pull request May 7, 2023

py-tensorflow: allow building for arm

114894d

See: #15397

essandess deleted the py-tensorflow branch May 7, 2023 19:36

mascguy mentioned this pull request May 9, 2023

bazel: update to 6.2.0 (minor LTS release) #18601

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

py-tensorflow: Update to version 2.12.0, Add Python 310 #15397

py-tensorflow: Update to version 2.12.0, Add Python 310 #15397

essandess commented Jul 18, 2022 •

edited

macportsbot commented Jul 18, 2022

mascguy commented Jul 20, 2022

essandess commented Jul 20, 2022

essandess commented Jul 30, 2022

mascguy commented Sep 2, 2022

essandess commented Sep 17, 2022

reneeotten commented May 7, 2023

mascguy commented May 7, 2023

mascguy commented May 7, 2023

mascguy commented May 7, 2023

mascguy commented May 7, 2023

catap commented May 7, 2023

mascguy commented May 7, 2023

catap commented May 7, 2023

mascguy commented May 7, 2023

mascguy commented May 7, 2023

mascguy commented May 7, 2023

mascguy commented May 7, 2023 •

edited

catap commented May 7, 2023

mascguy commented May 7, 2023

essandess commented May 7, 2023

mascguy commented May 8, 2023 •

edited

catap commented May 8, 2023

mascguy commented May 8, 2023 •

edited

cjones051073 commented May 9, 2023 •

edited

mascguy commented May 9, 2023 •

edited

catap commented May 9, 2023

py-tensorflow: Update to version 2.12.0, Add Python 310 #15397

py-tensorflow: Update to version 2.12.0, Add Python 310 #15397

Conversation

essandess commented Jul 18, 2022 • edited

Description

Type(s)

Tested on

Verification

macportsbot commented Jul 18, 2022

mascguy commented Jul 20, 2022

essandess commented Jul 20, 2022

essandess commented Jul 30, 2022

mascguy commented Sep 2, 2022

essandess commented Sep 17, 2022

reneeotten commented May 7, 2023

mascguy commented May 7, 2023

mascguy commented May 7, 2023

mascguy commented May 7, 2023

mascguy commented May 7, 2023

catap commented May 7, 2023

mascguy commented May 7, 2023

catap commented May 7, 2023

mascguy commented May 7, 2023

mascguy commented May 7, 2023

mascguy commented May 7, 2023

mascguy commented May 7, 2023 • edited

catap commented May 7, 2023

mascguy commented May 7, 2023

essandess commented May 7, 2023

mascguy commented May 8, 2023 • edited

catap commented May 8, 2023

mascguy commented May 8, 2023 • edited

cjones051073 commented May 9, 2023 • edited

mascguy commented May 9, 2023 • edited

catap commented May 9, 2023

essandess commented Jul 18, 2022 •

edited

mascguy commented May 7, 2023 •

edited

mascguy commented May 8, 2023 •

edited

mascguy commented May 8, 2023 •

edited

cjones051073 commented May 9, 2023 •

edited

mascguy commented May 9, 2023 •

edited