Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Apple Silicon support. #756

Merged
merged 7 commits into from
Dec 3, 2021
Merged

Conversation

sun1638650145
Copy link
Contributor

Description

Add Apple Silicon support. By installing tensorflow-macos 2.6.0 and tensorflow-metal 0.2.0, TF.Text can be compiled from source code(Currently Apple does not provide a preview version of tensorflow-macos and tf-nightly, so TF.Text can only build a stable version.).

Fixes #538

@google-cla google-cla bot added the cla: yes label Oct 31, 2021
Copy link
Member

@broken broken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR! There's been a lot of interest for this. :)

I'm not sure if we'll be able to update our builds to publish these to pypi yet, but this will at least allow others to build it themselves.

oss_scripts/configure.sh Show resolved Hide resolved
oss_scripts/pip_package/setup.py Outdated Show resolved Hide resolved
@sun1638650145
Copy link
Contributor Author

If the TensorFlow team and the community think that it is a better choice to continue using the standard version of TF on the x86-64 architecture Mac, I can also add the judgment of the architecture at compile time.

However, in general, I think it is more important for Mac developers to use GPU acceleration than high compatibility.

Copy link
Member

@broken broken left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for the delayed reply, I've been busy getting some last items ready for the 2.7 release.

@TeoZosa
Copy link

TeoZosa commented Nov 14, 2021

@sun1638650145 were you able to compile via bazel 3.7.2 on the M1 or did you find some way to get past the issues running > 4.1 versions of bazel (i.e., #638)? I'm stuck here and am stumped.

@sun1638650145
Copy link
Contributor Author

@TeoZosa I compiled the source code of bazel 3.7.2 because bazel 3.7.2 did not provide a pre-compiled package for M1. Currently compiling Tensorflow from source code must use bazel of 3.7.x, and bazel 4.x should not currently work.

@TeoZosa
Copy link

TeoZosa commented Nov 15, 2021

@TeoZosa I compiled the source code of bazel 3.7.2 because bazel 3.7.2 did not provide a pre-compiled package for M1. Currently compiling Tensorflow from source code must use bazel of 3.7.x, and bazel 4.x should not currently work.

Ah I was afraid you'd say that (compiling Bazel has been a pain). Thank you!

@sun1638650145
Copy link
Contributor Author

@TeoZosa Yes, Bazel officially recommends the use of pre-compiled packages.If you need it, I saved a compiled exec file (if you are using M1, you only need to add it to the environment variable to use).

@TeoZosa
Copy link

TeoZosa commented Nov 21, 2021

@TeoZosa Yes, Bazel officially recommends the use of pre-compiled packages.If you need it, I saved a compiled exec file (if you are using M1, you only need to add it to the environment variable to use).

Hey @sun1638650145, sorry for getting back so late. I actually figured out how to get it built locally, but I appreciate the offer! Fingers crossed your PR gets merged in soon :)

@hxssgaa
Copy link

hxssgaa commented Nov 23, 2021

Hey @sun1638650145, sorry for getting back so late. I actually figured out how to get it built locally, but I appreciate the offer! Fingers crossed your PR gets merged in soon :)

Could you provide the compiled exec file? I'm also stuck here.

@TeoZosa
Copy link

TeoZosa commented Nov 23, 2021

Could you provide the compiled exec file? I'm also stuck here.

Was that for me or @sun1638650145 ?

FYI In case it's useful to anyone, here are the source build instructions I used:

@hxssgaa
Copy link

hxssgaa commented Nov 23, 2021

Could you provide the compiled exec file? I'm also stuck here.

Was that for me or @sun1638650145 ?

FYI In case it's useful to anyone, here are the source build instructions I used:

Thanks, I already compiled the bazel 3.7.2 by myself without problems. But I still couldn't build tensorflow_text from source. I got the following error after executing ./oss_scripts/run_build.sh with the compiled bazel 3.7.2:

ERROR: /Users/hxssgaa/Documents/Developer/ML/text/tensorflow_text/core/kernels/BUILD:627:14: no such package '@com_google_sentencepiece//src': BUILD file not found in directory 'src' of external repository @com_google_sentencepiece. Add a BUILD file to a directory to mark it as a package. and referenced by '//tensorflow_text/core/kernels:sentencepiece_kernels'
ERROR: /Users/hxssgaa/Documents/Developer/ML/text/tensorflow_text/core/kernels/BUILD:627:14: no such package '@com_google_sentencepiece//src': BUILD file not found in directory 'src' of external repository @com_google_sentencepiece. Add a BUILD file to a directory to mark it as a package. and referenced by '//tensorflow_text/core/kernels:sentencepiece_kernels'
ERROR: /Users/hxssgaa/Documents/Developer/ML/text/tensorflow_text/core/kernels/BUILD:627:14: no such package '@com_google_sentencepiece//src': BUILD file not found in directory 'src' of external repository @com_google_sentencepiece. Add a BUILD file to a directory to mark it as a package. and referenced by '//tensorflow_text/core/kernels:sentencepiece_kernels'
ERROR: Analysis of target '//oss_scripts/pip_package:build_pip_package' failed; build aborted: Analysis failed

@TeoZosa
Copy link

TeoZosa commented Nov 23, 2021

Thanks, I already compiled the bazel 3.7.2 by myself without problems. But I still couldn't build tensorflow_text from source. I got the following error after executing ./oss_scripts/run_build.sh with the compiled bazel 3.7.2:

ERROR: /Users/hxssgaa/Documents/Developer/ML/text/tensorflow_text/core/kernels/BUILD:627:14: no such package '@com_google_sentencepiece//src': BUILD file not found in directory 'src' of external repository @com_google_sentencepiece. Add a BUILD file to a directory to mark it as a package. and referenced by '//tensorflow_text/core/kernels:sentencepiece_kernels'
ERROR: /Users/hxssgaa/Documents/Developer/ML/text/tensorflow_text/core/kernels/BUILD:627:14: no such package '@com_google_sentencepiece//src': BUILD file not found in directory 'src' of external repository @com_google_sentencepiece. Add a BUILD file to a directory to mark it as a package. and referenced by '//tensorflow_text/core/kernels:sentencepiece_kernels'
ERROR: /Users/hxssgaa/Documents/Developer/ML/text/tensorflow_text/core/kernels/BUILD:627:14: no such package '@com_google_sentencepiece//src': BUILD file not found in directory 'src' of external repository @com_google_sentencepiece. Add a BUILD file to a directory to mark it as a package. and referenced by '//tensorflow_text/core/kernels:sentencepiece_kernels'
ERROR: Analysis of target '//oss_scripts/pip_package:build_pip_package' failed; build aborted: Analysis failed

Ahh! Which branch are you building from? I had the same errors when building from an old version of master where a sentencepiece version bump broke the build, I had to checkout a >2.7.0 release branch (though that looks like it should be fixed in the new master).

@hxssgaa
Copy link

hxssgaa commented Nov 23, 2021

I'm building from the master branch in this repo: https://github.com/sun1638650145/text#build-from-source-steps. What exact branch you checkout fix the issue?

@TeoZosa
Copy link

TeoZosa commented Nov 23, 2021

I'm building from the master branch in this repo: https://github.com/sun1638650145/text#build-from-source-steps. What exact branch you checkout fix the issue?

I built from the tensorflow/text v2.7.3 tag .

Note: since @sun1638650145 's PR hasn't been merged yet, I had to manually patch the platform extension in build_pip_package.sh

@hxssgaa
Copy link

hxssgaa commented Nov 23, 2021

I'm building from the master branch in this repo: https://github.com/sun1638650145/text#build-from-source-steps. What exact branch you checkout fix the issue?

I built from the tensorflow/text 2.7.3 tag note: since @sun1638650145 's PR hasn't been merged yet, I had to manually patch the platform extension in build_pip_package.sh

You mean to replace macosx-10.9-x86_64 with macosx-11.0-arm64. Unfortunately I still got the above error after replacing the plat_name. Any clue for the problem?

@TeoZosa
Copy link

TeoZosa commented Nov 23, 2021

You mean to replace macosx-10.9-x86_64 with macosx-11.0-arm64. Unfortunately I still got the above error after replacing the plat_name.

I should have mentioned, that part is only so you can install the pip package after.

Any clue for the problem?

Did you try building from the v2.7.3 tag? That should do it (the sentencepiece files from master are different from those that get used for releases).

@hxssgaa
Copy link

hxssgaa commented Nov 23, 2021

You mean to replace macosx-10.9-x86_64 with macosx-11.0-arm64. Unfortunately I still got the above error after replacing the plat_name.

I should have mentioned, that part is only so you can install the pip package after.

Any clue for the problem?

Did you try building from the v2.7.3 tag? That should do it (the sentencepiece files from master are different from those that get used for releases).

I see, thanks. However I got new errors building from the v2.7.3 tag below:

ERROR: /Users/hxssgaa/Documents/Developer/ML/text/tensorflow_text/core/kernels/BUILD:173:14: C++ compilation of rule '//tensorflow_text/core/kernels:fast_wordpiece_tokenizer' failed (Exit 1): wrapped_clang failed: error executing command external/local_config_cc/wrapped_clang '-D_FORTIFY_SOURCE=1' -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG ... (remaining 54 argument(s) skipped)
tensorflow_text/core/kernels/fast_wordpiece_tokenizer.cc:25:10: fatal error: 'tensorflow/lite/kernels/shim/status_macros.h' file not found
#include "tensorflow/lite/kernels/shim/status_macros.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
Target //oss_scripts/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.

@TeoZosa
Copy link

TeoZosa commented Nov 23, 2021

I see, thanks. However I got new errors building from the v2.7.3 tag below:

ERROR: /Users/hxssgaa/Documents/Developer/ML/text/tensorflow_text/core/kernels/BUILD:173:14: C++ compilation of rule '//tensorflow_text/core/kernels:fast_wordpiece_tokenizer' failed (Exit 1): wrapped_clang failed: error executing command external/local_config_cc/wrapped_clang '-D_FORTIFY_SOURCE=1' -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG ... (remaining 54 argument(s) skipped)
tensorflow_text/core/kernels/fast_wordpiece_tokenizer.cc:25:10: fatal error: 'tensorflow/lite/kernels/shim/status_macros.h' file not found
#include "tensorflow/lite/kernels/shim/status_macros.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
Target //oss_scripts/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.

Which version of tensorflow are you using? FWIW, I also built tensorflow v2.7.0 from source and used that when building tensorflow-text v2.7.3

@hxssgaa
Copy link

hxssgaa commented Nov 23, 2021

I see, thanks. However I got new errors building from the v2.7.3 tag below:

ERROR: /Users/hxssgaa/Documents/Developer/ML/text/tensorflow_text/core/kernels/BUILD:173:14: C++ compilation of rule '//tensorflow_text/core/kernels:fast_wordpiece_tokenizer' failed (Exit 1): wrapped_clang failed: error executing command external/local_config_cc/wrapped_clang '-D_FORTIFY_SOURCE=1' -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG ... (remaining 54 argument(s) skipped)
tensorflow_text/core/kernels/fast_wordpiece_tokenizer.cc:25:10: fatal error: 'tensorflow/lite/kernels/shim/status_macros.h' file not found
#include "tensorflow/lite/kernels/shim/status_macros.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
Target //oss_scripts/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.

Which version of tensorflow are you using? FWIW, I also built tensorflow v2.7.0 from source and used that when building tensorflow-text v2.7.3

I'm using tensorflow-macos-2.6.0. I will try to build tensorflow v2.7.0 then. It looks like it's very complicated to install tensorflow_text. 😂

@hxssgaa
Copy link

hxssgaa commented Nov 23, 2021

I see, thanks. However I got new errors building from the v2.7.3 tag below:

ERROR: /Users/hxssgaa/Documents/Developer/ML/text/tensorflow_text/core/kernels/BUILD:173:14: C++ compilation of rule '//tensorflow_text/core/kernels:fast_wordpiece_tokenizer' failed (Exit 1): wrapped_clang failed: error executing command external/local_config_cc/wrapped_clang '-D_FORTIFY_SOURCE=1' -fstack-protector -fcolor-diagnostics -Wall -Wthread-safety -Wself-assign -fno-omit-frame-pointer -g0 -O2 '-D_FORTIFY_SOURCE=1' -DNDEBUG ... (remaining 54 argument(s) skipped)
tensorflow_text/core/kernels/fast_wordpiece_tokenizer.cc:25:10: fatal error: 'tensorflow/lite/kernels/shim/status_macros.h' file not found
#include "tensorflow/lite/kernels/shim/status_macros.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
Target //oss_scripts/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.

Which version of tensorflow are you using? FWIW, I also built tensorflow v2.7.0 from source and used that when building tensorflow-text v2.7.3

Have you done any patching for tensorflow v2.7.0 for building as well? I got the following error when building tensorflow with bazel build //tensorflow/tools/pip_package:build_pip_package

ERROR: /Users/hxssgaa/Documents/Developer/ML/tensorflow/tensorflow/compiler/tf2xla/cc/BUILD:9:21: Executing genrule //tensorflow/compiler/tf2xla/cc:xla_ops_gen_genrule failed (Aborted): bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped)
dyld[93221]: symbol not found in flat namespace '_ChaCha20_ctr32'
/bin/bash: line 1: 93221 Abort trap: 6           bazel-out/host/bin/tensorflow/compiler/tf2xla/cc/ops/xla_ops_gen_cc bazel-out/darwin-opt/bin/tensorflow/compiler/tf2xla/cc/ops/xla_ops.h bazel-out/darwin-opt/bin/tensorflow/compiler/tf2xla/cc/ops/xla_ops.cc 0 ,
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 242.966s, Critical Path: 32.20s
INFO: 3886 processes: 604 internal, 3282 local.
FAILED: Build did NOT complete successfully

@TeoZosa
Copy link

TeoZosa commented Nov 23, 2021

Have you done any patching for tensorflow v2.7.0 for building as well? I got the following error when building tensorflow with bazel build //tensorflow/tools/pip_package:build_pip_package

ERROR: /Users/hxssgaa/Documents/Developer/ML/tensorflow/tensorflow/compiler/tf2xla/cc/BUILD:9:21: Executing genrule //tensorflow/compiler/tf2xla/cc:xla_ops_gen_genrule failed (Aborted): bash failed: error executing command /bin/bash -c ... (remaining 1 argument(s) skipped)
dyld[93221]: symbol not found in flat namespace '_ChaCha20_ctr32'
/bin/bash: line 1: 93221 Abort trap: 6           bazel-out/host/bin/tensorflow/compiler/tf2xla/cc/ops/xla_ops_gen_cc bazel-out/darwin-opt/bin/tensorflow/compiler/tf2xla/cc/ops/xla_ops.h bazel-out/darwin-opt/bin/tensorflow/compiler/tf2xla/cc/ops/xla_ops.cc 0 ,
Target //tensorflow/tools/pip_package:build_pip_package failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 242.966s, Critical Path: 32.20s
INFO: 3886 processes: 604 internal, 3282 local.
FAILED: Build did NOT complete successfully

Hm. That's an interesting one. Are you compiling with XLA?

To compile TF I just used the default values for ./configure and built with:

sudo bazel build --config=macos_arm64 //tensorflow/tools/pip_package:build_pip_package

(see my here for the full build instructions)

@hxssgaa
Copy link

hxssgaa commented Nov 23, 2021

my here for the full build instructions

I tried both following your script and installing the tensorflow by myself using --config=macos_arm64 but still got the above error:

ERROR: /Users/hxssgaa/Documents/Developer/ML/tensorflow/tensorflow/python/BUILD:806:29: Executing genrule //tensorflow/python:sdca_ops_pygenrule failed (Aborted): bash failed: error executing command /bin/bash bazel-out/darwin_arm64-opt/bin/tensorflow/python/sdca_ops_pygenrule.genrule_script.sh
dyld[33329]: symbol not found in flat namespace '_ChaCha20_ctr32'

Not sure why.

@yuyuan20
Copy link

yuyuan20 commented Feb 3, 2022

Thank you @sun1638650145. As per your instructions, I have created a new conda environment which uses tensorflow-macos.

# packages in environment at /opt/homebrew/Caskroom/miniforge/base/envs/ml-env:
#
# Name                    Version                   Build  Channel
tensorflow-deps           2.7.0                         0    apple
tensorflow-estimator      2.7.0                    pypi_0    pypi
tensorflow-macos          2.7.0                    pypi_0    pypi
tensorflow-metal          0.3.0                    pypi_0    pypi

--------------------------------------------------------
machine: Macbook Air M1 (2021) running macOS Monterey
python version: 3.9.10
pip version: 22.0.2

I tried replacing the file as @broken mentioned in #823 but the output is still the same:

++ uname -s
+ osname=Darwin
+ [[ Darwin == \D\a\r\w\i\n ]]
+ sed -i '' 's/".so"/".dylib"/' tensorflow_text/tftext.bzl
++ find tensorflow_text/python -type f
+ perl -pi -e 's/(load_library.load_op_library.*)\.so'\''/$1.dylib'\''/' tensorflow_text/python/keras/layers/tokenization_layers.py tensorflow_text/python/keras/layers/__init__.py tensorflow_text/python/keras/layers/todense.py tensorflow_text/python/keras/layers/todense_test.py tensorflow_text/python/keras/layers/tokenization_layers_test.py tensorflow_text/python/keras/__init__.py tensorflow_text/python/metrics/text_similarity_metric_ops.py tensorflow_text/python/metrics/__init__.py tensorflow_text/python/metrics/text_similarity_metric_ops_test.py tensorflow_text/python/.DS_Store tensorflow_text/python/ragged/ragged_tensor_to_tensor_test.py tensorflow_text/python/__init__.py tensorflow_text/python/numpy/viterbi_decode.py tensorflow_text/python/numpy/__init__.py tensorflow_text/python/numpy/viterbi_decode_test.py tensorflow_text/python/benchmarks/ops_benchmarks.py tensorflow_text/python/benchmarks/test_data/uncased_L-12_H-768_A-12/vocab.txt tensorflow_text/python/benchmarks/__init__.py tensorflow_text/python/benchmarks/tokenizers_benchmarks.py tensorflow_text/python/benchmarks/benchmark_utils.py tensorflow_text/python/ops/string_ops.py tensorflow_text/python/ops/viterbi_constrained_sequence_op_test.py tensorflow_text/python/ops/sentencepiece_tokenizer.py tensorflow_text/python/ops/normalize_ops_test.py tensorflow_text/python/ops/trimmer_ops_test.py tensorflow_text/python/ops/masking_ops.py tensorflow_text/python/ops/pointer_ops.py tensorflow_text/python/ops/hub_module_tokenizer_test.py tensorflow_text/python/ops/regex_split_ops_test.py tensorflow_text/python/ops/bert_tokenizer_test.py tensorflow_text/python/ops/fast_wordpiece_tokenizer.py tensorflow_text/python/ops/pad_model_inputs_ops.py tensorflow_text/python/ops/item_selector_ops.py tensorflow_text/python/ops/state_based_sentence_breaker_op.py tensorflow_text/python/ops/split_merge_from_logits_tokenizer.py tensorflow_text/python/ops/pad_along_dimension_op_test.py tensorflow_text/python/ops/segment_combiner_ops_test.py tensorflow_text/python/ops/unicode_char_tokenizer_test.py tensorflow_text/python/ops/test_data/test_wp_en_vocab.txt tensorflow_text/python/ops/test_data/test_oss_model.model tensorflow_text/python/ops/test_data/fast_wordpiece_tokenizer_model.fb tensorflow_text/python/ops/test_data/segmenter_hub_module/variables/variables.data-00000-of-00001 tensorflow_text/python/ops/test_data/segmenter_hub_module/variables/variables.index tensorflow_text/python/ops/test_data/segmenter_hub_module/saved_model.pb tensorflow_text/python/ops/test_data/segmenter_hub_module/tfhub_module.pb tensorflow_text/python/ops/test_data/fast_wordpiece_README.google.txt tensorflow_text/python/ops/wordpiece_tokenizer_test.py tensorflow_text/python/ops/ngrams_op_test.py tensorflow_text/python/ops/greedy_constrained_sequence_op_test.py tensorflow_text/python/ops/splitter.py tensorflow_text/python/ops/viterbi_constrained_sequence_op.py tensorflow_text/python/ops/__init__.py tensorflow_text/python/ops/split_merge_from_logits_tokenizer_test.py tensorflow_text/python/ops/tokenization.py tensorflow_text/python/ops/span_alignment_op_test.py tensorflow_text/python/ops/whitespace_tokenizer_test.py tensorflow_text/python/ops/split_merge_tokenizer.py tensorflow_text/python/ops/bert_tokenizer.py tensorflow_text/python/ops/hub_module_splitter_test.py tensorflow_text/python/ops/wordshape_ops_test.py tensorflow_text/python/ops/span_overlaps_op_test.py tensorflow_text/python/ops/sentence_breaking_ops.py tensorflow_text/python/ops/wordpiece_tokenizer.py tensorflow_text/python/ops/sliding_window_op_test.py tensorflow_text/python/ops/trimmer_ops.py tensorflow_text/python/ops/segment_combiner_ops.py tensorflow_text/python/ops/unicode_script_tokenizer.py tensorflow_text/python/ops/normalize_ops.py tensorflow_text/python/ops/hub_module_tokenizer.py tensorflow_text/python/ops/whitespace_tokenizer.py tensorflow_text/python/ops/mst_ops.py tensorflow_text/python/ops/state_based_sentence_breaker_op_test.py tensorflow_text/python/ops/sliding_window_op.py tensorflow_text/python/ops/sentence_breaking_ops_test.py tensorflow_text/python/ops/masking_ops_test.py tensorflow_text/python/ops/greedy_constrained_sequence_op.py tensorflow_text/python/ops/unicode_script_tokenizer_test.py tensorflow_text/python/ops/gather_with_default_op_test.py tensorflow_text/python/ops/split_merge_tokenizer_test.py tensorflow_text/python/ops/create_feature_bitmask_op.py tensorflow_text/python/ops/pad_along_dimension_op.py tensorflow_text/python/ops/create_feature_bitmask_op_test.py tensorflow_text/python/ops/ngrams_op.py tensorflow_text/python/ops/sentencepiece_tokenizer_test.py tensorflow_text/python/ops/item_selector_ops_test.py tensorflow_text/python/ops/regex_split_ops.py tensorflow_text/python/ops/coerce_to_valid_utf8_op_test.py tensorflow_text/python/ops/hub_module_splitter.py tensorflow_text/python/ops/unicode_char_tokenizer.py tensorflow_text/python/ops/wordshape_ops.py tensorflow_text/python/ops/fast_wordpiece_tokenizer_test.py tensorflow_text/python/ops/pad_model_inputs_ops_test.py tensorflow_text/python/ops/mst_ops_test.py
+ export CC_OPT_FLAGS=-mavx
+ CC_OPT_FLAGS=-mavx
+ source oss_scripts/configure.sh
+++ uname -s
+++ tr A-Z a-z
++ osname=darwin
++ echo darwin
darwin
++ '[' -e .bazelrc ']'
++ rm .bazelrc
+++ pip show tensorflow
WARNING: Package(s) not found: tensorflow
++ [[ '' == *tensorflow* ]]
+++ pip show tensorflow-macos
++ [[ Name: tensorflow-macos
Version: 2.7.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.ml-envtensorflow.org/
Author: Google Inc.
Author-email: packages@tensorflow.org
License: Apache 2.0
Location: /opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages
Requires: absl-py, astunparse, flatbuffers, gast, google-pasta, grpcio, h5py, keras, keras-preprocessing, libclang, numpy, opt-einsum, protobuf, six, tensorboard, tensorflow-estimator, termcolor, typing-extensions, wheel, wrapt
Required-by:  == *tensorflow-macos* ]]
++ echo 'Using installed tensorflow.'
Using installed tensorflow.
++ is_windows
++ [[ darwin =~ msys_nt*|mingw*|cygwin*|uwin* ]]
++ write_to_bazelrc 'build:manylinux2010 --crosstool_top=@ubuntu18.04-gcc7_manylinux2010-cuda11.2-cudnn8.1-tensorrt7.2_config_cuda//crosstool:toolchain'
++ echo 'build:manylinux2010 --crosstool_top=@ubuntu18.04-gcc7_manylinux2010-cuda11.2-cudnn8.1-tensorrt7.2_config_cuda//crosstool:toolchain'
++ write_to_bazelrc 'build --spawn_strategy=standalone'
++ echo 'build --spawn_strategy=standalone'
++ write_to_bazelrc 'build --strategy=Genrule=standalone'
++ echo 'build --strategy=Genrule=standalone'
++ write_to_bazelrc 'build -c opt'
++ echo 'build -c opt'
++ write_to_bazelrc 'build --define=framework_shared_object=true'
++ echo 'build --define=framework_shared_object=true'
++ write_to_bazelrc 'build --experimental_repo_remote_exec'
++ echo 'build --experimental_repo_remote_exec'
++ write_to_bazelrc 'build --incompatible_blacklisted_protos_requires_proto_info=false'
++ echo 'build --incompatible_blacklisted_protos_requires_proto_info=false'
++ write_to_bazelrc 'build --cxxopt=-std=c++14'
++ echo 'build --cxxopt=-std=c++14'
++ write_to_bazelrc 'build --host_cxxopt=-std=c++14'
++ echo 'build --host_cxxopt=-std=c++14'
++ write_to_bazelrc 'build:android --crosstool_top=//external:android/crosstool'
++ echo 'build:android --crosstool_top=//external:android/crosstool'
++ write_to_bazelrc 'build:android --host_crosstool_top=@bazel_tools//tools/cpp:toolchain'
++ echo 'build:android --host_crosstool_top=@bazel_tools//tools/cpp:toolchain'
++ write_to_bazelrc 'build:android --action_env TF_HEADER_DIR=""'
++ echo 'build:android --action_env TF_HEADER_DIR=""'
++ write_to_bazelrc 'build:android --action_env TF_SHARED_LIBRARY_DIR=""'
++ echo 'build:android --action_env TF_SHARED_LIBRARY_DIR=""'
++ write_to_bazelrc 'build:android --action_env TF_SHARED_LIBRARY_NAME=""'
++ echo 'build:android --action_env TF_SHARED_LIBRARY_NAME=""'
++ write_to_bazelrc 'build:android_arm --config=android'
++ echo 'build:android_arm --config=android'
++ write_to_bazelrc 'build:android_arm --cpu=armeabi-v7a'
++ echo 'build:android_arm --cpu=armeabi-v7a'
++ write_to_bazelrc 'build:android_arm --fat_apk_cpu=armeabi-v7a'
++ echo 'build:android_arm --fat_apk_cpu=armeabi-v7a'
++ write_to_bazelrc 'build:android_arm64 --config=android'
++ echo 'build:android_arm64 --config=android'
++ write_to_bazelrc 'build:android_arm64 --cpu=arm64-v8a'
++ echo 'build:android_arm64 --cpu=arm64-v8a'
++ write_to_bazelrc 'build:android_arm64 --fat_apk_cpu=arm64-v8a'
++ echo 'build:android_arm64 --fat_apk_cpu=arm64-v8a'
++ write_to_bazelrc 'build:android_x86 --config=android'
++ echo 'build:android_x86 --config=android'
++ write_to_bazelrc 'build:android_x86 --cpu=x86'
++ echo 'build:android_x86 --cpu=x86'
++ write_to_bazelrc 'build:android_x86 --fat_apk_cpu=x86'
++ echo 'build:android_x86 --fat_apk_cpu=x86'
++ write_to_bazelrc 'build:android_x86_64 --config=android'
++ echo 'build:android_x86_64 --config=android'
++ write_to_bazelrc 'build:android_x86_64 --cpu=x86_64'
++ echo 'build:android_x86_64 --cpu=x86_64'
++ write_to_bazelrc 'build:android_x86_64 --fat_apk_cpu=x86_64'
++ echo 'build:android_x86_64 --fat_apk_cpu=x86_64'
++ is_windows
++ [[ darwin =~ msys_nt*|mingw*|cygwin*|uwin* ]]
++ which python
++ grep -q python
++ installed_python=python
++ TF_CFLAGS=($($installed_python -c "import tensorflow as tf; print(' '.join(tf.sysconfig.get_compile_flags()))" | awk '{print $1}'))
+++ python -c 'import tensorflow as tf; print('\'' '\''.join(tf.sysconfig.get_compile_flags()))'
+++ awk '{print $1}'
++ TF_LFLAGS=($($installed_python -c "import tensorflow as tf; print(' '.join(tf.sysconfig.get_link_flags()))" | awk '{print $1}'))
+++ python -c 'import tensorflow as tf; print('\'' '\''.join(tf.sysconfig.get_link_flags()))'
+++ awk '{print $1}'
++ TF_LFLAGS_2=($($installed_python -c "import tensorflow as tf; print(' '.join(tf.sysconfig.get_link_flags()))" | awk '{print $2}'))
+++ python -c 'import tensorflow as tf; print('\'' '\''.join(tf.sysconfig.get_link_flags()))'
+++ awk '{print $2}'
+++ python -c 'import tensorflow as tf; print(tf.sysconfig.CXX11_ABI_FLAG)'
++ TF_ABIFLAG=0
++ HEADER_DIR=/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow/include
++ SHARED_LIBRARY_DIR=/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow
+++ echo -ltensorflow_framework.2
+++ rev
+++ cut -d: -f1
+++ rev
++ SHARED_LIBRARY_NAME=-ltensorflow_framework.2
++ is_macos
++ [[ darwin == \d\a\r\w\i\n ]]
++ SHARED_LIBRARY_NAME=libtensorflow_framework.dylib
++ is_windows
++ [[ darwin =~ msys_nt*|mingw*|cygwin*|uwin* ]]
++ export LD_LIBRARY_PATH=:/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow
++ LD_LIBRARY_PATH=:/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow
++ write_action_env_to_bazelrc TF_HEADER_DIR /opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow/include
++ write_to_bazelrc 'build --action_env TF_HEADER_DIR="/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow/include"'
++ echo 'build --action_env TF_HEADER_DIR="/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow/include"'
++ write_action_env_to_bazelrc TF_SHARED_LIBRARY_DIR /opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow
++ write_to_bazelrc 'build --action_env TF_SHARED_LIBRARY_DIR="/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow"'
++ echo 'build --action_env TF_SHARED_LIBRARY_DIR="/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow"'
++ write_action_env_to_bazelrc TF_SHARED_LIBRARY_NAME libtensorflow_framework.dylib
++ write_to_bazelrc 'build --action_env TF_SHARED_LIBRARY_NAME="libtensorflow_framework.dylib"'
++ echo 'build --action_env TF_SHARED_LIBRARY_NAME="libtensorflow_framework.dylib"'
++ write_action_env_to_bazelrc TF_CXX11_ABI_FLAG 0
++ write_to_bazelrc 'build --action_env TF_CXX11_ABI_FLAG="0"'
++ echo 'build --action_env TF_CXX11_ABI_FLAG="0"'
+ source oss_scripts/prepare_tf_dep.sh
++ set -e
++ set -x
++ which python
++ grep -q python
++ installed_python=python
+++ python -c 'import tensorflow as tf; print(tf.__version__)'
++ tf_version=2.7.0
++ echo 2.7.0
2.7.0
++ sed -i 's/project_version = '\''REPLACE_ME'\''/project_version = '\''2.7.0'\''/' oss_scripts/pip_package/setup.nightly.py
sed: 1: "oss_scripts/pip_package ...": invalid command code o

No .whl file getting generated whatsoever. Is there something I am missing?

@yulsa
Copy link

yulsa commented Feb 4, 2022

hi, I have what looks like exact same issue as yuyuan20

environment:

  • apple M1
  • MacOS Montenerry 12.1
  • python 3.9
  • tensorflow-macos==2.7.0 (installed via Conda)

cloned repo with git clone https://github.com/tensorflow/text.git and then ran ./oss_scripts/run_build.sh in the environment

the file outputs very long list, same as above, and at the end it breaks with:

...
++ installed_python=python
+++ python -c 'import tensorflow as tf; print(tf.__version__)'
++ tf_version=2.7.0
++ echo 2.7.0
2.7.0
++ sed -i 's/project_version = '\''REPLACE_ME'\''/project_version = '\''2.7.0'\''/' oss_scripts/pip_package/setup.nightly.py
sed: 1: "oss_scripts/pip_package ...": invalid command code o

I verified that contents of prepare_tf_dep.sh is as specified in #823, but the error remains the same

@yulsa
Copy link

yulsa commented Feb 4, 2022

some additional findings after trying to solve the above:

  • based on some googling, sed behaves differently on Mac sometimes. added -e and it seemed to work

  • grep also behaves differently on the Mac OS, as it doesn't have -oP and was thus breaking in prepare_tf_dep.sh . To bypass that installed GNU grep via Brew and added to path.

  • now the code stalls in prepare_tf_dep.sh at around short_commit_sha= as no value gets passed. I tested directly in the browser, and python -c 'import tensorflow as tf; print(tf.__git_version__)' actually returns 'unknown'. Is it because I'm using the tensorflow-macos? Is there other way to extract whatever value needs to go in here? at the moment the build does not continue past

++ echo python
python
+++ python -c 'import tensorflow as tf; print(tf.__git_version__)'
+++ tail -1
+++ grep -oP '(?<=-g)[0-9a-f]*$'
++ short_commit_sha=

Screen Shot 2022-02-04 at 12 56 23 AM

@yuyuan20
Copy link

yuyuan20 commented Feb 4, 2022

some additional findings after trying to solve the above:

  • based on some googling, sed behaves differently on Mac sometimes. added -e and it seemed to work

  • grep also behaves differently on the Mac OS, as it doesn't have -oP and was thus breaking in prepare_tf_dep.sh . To bypass that installed GNU grep via Brew and added to path.

  • now the code stalls in prepare_tf_dep.sh at around short_commit_sha= as no value gets passed. I tested directly in the browser, and python -c 'import tensorflow as tf; print(tf.__git_version__)' actually returns 'unknown'. Is it because I'm using the tensorflow-macos? Is there other way to extract whatever value needs to go in here? at the moment the build does not continue past

++ echo python
python
+++ python -c 'import tensorflow as tf; print(tf.__git_version__)'
+++ tail -1
+++ grep -oP '(?<=-g)[0-9a-f]*$'
++ short_commit_sha=

Screen Shot 2022-02-04 at 12 56 23 AM

Can you provide your edited sh file so that I can test on my machine and see if it does anything?

@yulsa
Copy link

yulsa commented Feb 4, 2022

Did a bit more tweaking to keep sed working as is -> installed GNU sed via brew and added it to path. So sed commands now pass without modification needed.

Added this line to confirm version vs git version check: version_check=$($installed_python -c 'import tensorflow as tf; print("\nversion: {}".format(tf.__version__)); print("git_version: {}".format(tf.__git_version__))')
Script doesn't really have edits, just checkpoints and comments, but here it is:

file: prepare_tf_dep.sh

#!/bin/bash
set -e  # fail and exit on any command erroring
set -x  # print evaluated commands

if (which python) | grep -q "python"; then
  installed_python="python"
elif (which python3) | grep -q "python3"; then
  installed_python="python3"
fi

# update setup.nightly.py with tf version
tf_version=$($installed_python -c 'import tensorflow as tf; print(tf.__version__)')
echo "$tf_version"

sed -i "s/project_version = 'REPLACE_ME'/project_version = '${tf_version}'/" oss_scripts/pip_package/setup.nightly.py
# getting sed error `invalid command code o`  Potential culprit is MAC using BSD Sed, and this being GNU?
# installed gsed via brew, added to path, to enable using GNU sed 

echo "---------- checkpoint 1 ----------"

# update __version__
sed -i "s/__version__ = .*\$/__version__ = \"${tf_version}\"/" tensorflow_text/__init__.py

echo "---------- checkpoint 2 ----------"

# Get commit sha of installed tensorflow
echo "$installed_python"

echo "---------- check tf version and git version used ----------"

version_check=$($installed_python -c 'import tensorflow as tf; print("\nversion: {}".format(tf.__version__)); print("git_version: {}".format(tf.__git_version__))')
#echo "$version_check"

short_commit_sha=$($installed_python -c 'import tensorflow as tf; print(tf.__git_version__)' | tail -1 | grep -oP '(?<=-g)[0-9a-f]*$')

echo "---------- checkpoint 3 ----------"


commit_sha=$(curl -SsL https://github.com/tensorflow/tensorflow/commit/${short_commit_sha} | grep sha-block | grep commit | sed -e 's/.*\([a-f0-9]\{40\}\).*/\1/')

echo "---------- checkpoint 4 ----------"

# Update TF dependency to installed tensorflow
sed -i "s/strip_prefix = \"tensorflow-2\.[0-9]\+\.[0-9]\+\(-rc[0-9]\+\)\?\",/strip_prefix = \"tensorflow-${commit_sha}\",/" WORKSPACE
sed -i "s|\"https://github.com/tensorflow/tensorflow/archive/v.\+\.zip\"|\"https://github.com/tensorflow/tensorflow/archive/${commit_sha}.zip\"|" WORKSPACE
prev_shasum=$(grep -A 1 -e "strip_prefix.*tensorflow-" WORKSPACE | tail -1 | awk -F '"' '{print $2}')
sed -i "s/sha256 = \"${prev_shasum}\",//" WORKSPACE

@yuyuan20
Copy link

yuyuan20 commented Feb 4, 2022

Now it throws a different error.

++ uname -s
+ osname=Darwin
+ [[ Darwin == \D\a\r\w\i\n ]]
+ gsed -i '' 's/".so"/".dylib"/' tensorflow_text/tftext.bzl
gsed: can't read s/".so"/".dylib"/: No such file or directory

Still can't figure out where the problem is :)

@yuyuan20
Copy link

yuyuan20 commented Feb 4, 2022

Now it throws a different error.

++ uname -s
+ osname=Darwin
+ [[ Darwin == \D\a\r\w\i\n ]]
+ gsed -i '' 's/".so"/".dylib"/' tensorflow_text/tftext.bzl
gsed: can't read s/".so"/".dylib"/: No such file or directory

Still can't figure out where the problem is :)

Yeah that was because I installed gnu-sed from homebrew, even though I had one in-built. Now the error is as same as before:
sed: 1: "oss_scripts/pip_package ...": invalid command code o

@yulsa
Copy link

yulsa commented Feb 4, 2022

Now it throws a different error.

++ uname -s
+ osname=Darwin
+ [[ Darwin == \D\a\r\w\i\n ]]
+ gsed -i '' 's/".so"/".dylib"/' tensorflow_text/tftext.bzl
gsed: can't read s/".so"/".dylib"/: No such file or directory

Still can't figure out where the problem is :)

I'm able to bypass this by going back to Mac sed and either adding ' ' or -e after -i. eg: sed -i '' "s/project_version = 'REPLACE_ME'/project_version = '${tf_version}'/" oss_scripts/pip_package/setup.nightly.py

did you figure out how to get the version of tensorflow git? if you run python -c 'import tensorflow as tf; print("\nversion: {}".format(tf.__version__)); print("git_version: {}".format(tf.__git_version__))' in command line (inside correct Conda environment) do you get any git version at all?

can't get past that point...

@broken
Copy link
Member

broken commented Feb 4, 2022

@yuyuan20 please continue this discussion on #823 as your issues are independent of this PR

@broken
Copy link
Member

broken commented Feb 4, 2022

I responded on the bug, but I'll add context here in case somebody else runs across it, as it does pertain to building against M1.

We added the script prepare_tf_dep.sh to the run_build script to make sure that manual builds of TF Text were built against the version of TF you have installed. This was especially important for those building against nightly, and has reduced confusion for those users.

However, in the case of tensorflow-macos, it seems like Apple does not populate the git_version variable, and there is no tensorflow-macos-nightly for users to build against. Thus, the script is obsolete in this case.

We'll update run_build to not run this script for these builds, but in the meantime, you can simply remove the line source oss_scripts/prepare_tf_dep.sh from oss_scripts/run_build.sh.

@sun1638650145
Copy link
Contributor Author

@yulsa @yuyuan20 Apple updated tensorflow-macos 2.8.0, I will build tensorflow-text 2.8.1 on this version and try to fix the bugs you have.

@yuyuan20
Copy link

yuyuan20 commented Feb 14, 2022

@yulsa @yuyuan20 Apple updated tensorflow-macos 2.8.0, I will build tensorflow-text 2.8.1 on this version and try to fix the bugs you have.

Thank you. Also please provide the link to the wheel so I do not have to go through the frustration to build it. I have successfully build it by directly downloading the wheel as @broken mentioned here #823 (comment)_ . The import and everything works fine but the code does not execute if tensorflow-metal(ver 0.3.0) is installed. So the only way to run this is to uninstall tensorflow-metal but that takes too much time to train a simple BERT model (approx 1 hour per epoch). I have reported this issue to tensorflow but they replied this issue is specific to macOS. Anyway, I have to see if the issue gets resolved by upgrading both tensorflow and text. Regards.

@maparent
Copy link

Did not have time to test yet, but I have built the wheel: https://idealoom.org/wheelhouse/tensorflow_text-2.8.1-cp39-cp39-macosx_11_0_arm64.whl

@sun1638650145
Copy link
Contributor Author

@yuyuan20 In fact, tensorflow-metal is a closed-source project built by Apple, and we have no way to fix the bug you have.

@yuyuan20
Copy link

@yuyuan20 In fact, tensorflow-metal is a closed-source project built by Apple, and we have no way to fix the bug you have.

Yeah I know. Just mentioned to check if anyone faced the same issue or if the issue is really related to tensorflow-metal.

@yuyuan20
Copy link

Did not have time to test yet, but I have built the wheel: https://idealoom.org/wheelhouse/tensorflow_text-2.8.1-cp39-cp39-macosx_11_0_arm64.whl

Installed and tried to import it but getting this error:

Traceback (most recent call last):
  File "/Users/yuyuan/Desktop/ml-env/Text Classifier/main.py", line 4, in <module>
    import tensorflow_text as text
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow_text/__init__.py", line 21, in <module>
    from tensorflow_text.python import keras
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow_text/python/keras/__init__.py", line 21, in <module>
    from tensorflow_text.python.keras.layers import *
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow_text/python/keras/layers/__init__.py", line 22, in <module>
    from tensorflow_text.python.keras.layers.tokenization_layers import *
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow_text/python/keras/layers/tokenization_layers.py", line 25, in <module>
    from tensorflow_text.python.ops import unicode_script_tokenizer
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow_text/python/ops/__init__.py", line 24, in <module>
    from tensorflow_text.python.ops.bert_tokenizer import BertTokenizer
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow_text/python/ops/bert_tokenizer.py", line 28, in <module>
    from tensorflow_text.python.ops import regex_split_ops
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow_text/python/ops/regex_split_ops.py", line 24, in <module>
    gen_regex_split_ops = load_library.load_op_library(resource_loader.get_path_to_datafile('_regex_split_ops.dylib'))
  File "/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow/python/framework/load_library.py", line 54, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: dlopen(/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow_text/python/ops/_regex_split_ops.dylib, 0x0006): tried: '/opt/homebrew/Caskroom/miniforge/base/envs/ml-env/lib/python3.9/site-packages/tensorflow_text/python/ops/_regex_split_ops.dylib' (no such file), '/usr/local/lib/_regex_split_ops.dylib' (no such file), '/usr/lib/_regex_split_ops.dylib' (no such file)

@maparent
Copy link

Thanks for the report, and apologies for not having at least tested import. There was indeed an issue with my wheel, and I rebuilt it. Please try again (with the same URL).

@sun1638650145
Copy link
Contributor Author

@yulsa @yuyuan20 I wrote a complete tutorial on building text. Of course, you can also download the wheel files directly, both Python 3.8 and Python 3.9 are provided. I re-created the environment, and your problem did occur. The solution is here.

@sun1638650145
Copy link
Contributor Author

sun1638650145 commented Feb 16, 2022

@legacyai I'm sorry it took so long to write the tutorial. Here's the link, hope it's still helpful.

@sun1638650145
Copy link
Contributor Author

I have written a complete build tutorial, along with pre-compiled wheel, which I hope can help those who need it.

@pz325
Copy link

pz325 commented Feb 17, 2022

The pre-compiled wheels have saved my days. Could a hero help make a tensorflow-text 2.6 pre-compiled wheel (cp39-cp39-macosx_11_0_arm64) please?

@pz325
Copy link

pz325 commented Feb 17, 2022

Also: could a hero make an aarch64-manylinux wheel please?

(cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64)

@sun1638650145
Copy link
Contributor Author

@pz325 I have a tensorflow_text-2.6.0-cp38-cp38-macosx_11_0_arm64.whl here. I don't know if it can help you. Because the new version of text updated the bazel dependency, I don't have the old version of bazel 3.7.2, so I need to recompile it, I'm not sure if I have time to do that this week. For the aarch64 version of the file, you can find the issue area, or you can sort out the problems you have compiled and submit a new issue. Only Apple Silicon is discussed here.

broken referenced this pull request Apr 19, 2022
The TensorFlow Text package requires use of a matching minor version of TensorFlow. In order to support Apple Silicon macs, Tensorflow Text chooses between the `tensorflow` and `tensorflow-macos` packages automatically based on system architecture. However, I believe that commit a7e1b5b accidentally wiped out this logic. Additionally, this commit failed to increment the minor version of the `tensorflow-macos` dependency.

This typically isn't a problem for users installing TensorFlow Text using `pip` on non-Apple Silicon devices, since `pip` just tries to find the best set of packages for the current system. However, this _is_ a problem for package managers like Poetry, which create a deterministic lock file allowing a similar set of dependencies to be installed across different systems, modulo any architectural differences. Since Poetry needs to resolve a combination of dependencies that will work for _any_ system, it eagerly tries to identify a version of `tensorflow-macos` that will work on Apple Silicon devices. However, since the minor versions of the `tensorflow` and `tensorflow-macos` dependencies don't match, it's unable to find an appropriate version of `keras` that both can use. As a result, installing `tensorflow-text (>=2.8.0)` using Poetry is currently impossible.

These changes restore the logic to automatically select between `tensorflow` and `tensorflow-macos` based on platform and architecture identifiers, and bump the version of the `tensorflow-macos` dependency to match the minor version of `tensorflow` and `tensorflow-text`.
@ashsha21
Copy link

@sun1638650145 Thanks for the precompiled wheels. It seems it is working :) Able to train BERT now. It is interesting one epochs is taking almost 20hrs . I do have lot of training data but with GPU I feel it is still long time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Can not build on Apple Silicon from source.