Split env library so that there is only one instance of Env::Default()
#43594
+37
−2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
3 tasks
mihaimaruseac
previously approved these changes
Sep 28, 2020
google-ml-butler
bot
added
kokoro:force-run
gbaned
added
ready to pull
|
This one needs a lot of changes internally since we also have a Google-only |
|
There is actually an issue with splitting things out like here. External packages using TF Bazel (not from pip package) are supposed to be able to get to |
2 tasks
While working on modular file systems, noticed that tensorflow's pip package includes two copies of `Env::Default()` in two shared objects libraries. One is in `libtensorflow_framework.so`, another is in `_pywrap_tensorflow_internal.so`. Because of that, `Env::Default()` could return different instance depending on when a modular file system is called. The Lookup vs. Register discrepancy may confuse the file system to believe the scheme (e.g., `s3`/`gcs`) is not registered (actually registered by another `Env::Default()`). The duplication of `Env::Default()` is caused by the fact that `tensorflow/core/platform:env` dependency is included in both `libtensorflow_framework.so` and `_pywrap_tensorflow_internal.so`. in bazel indirectly. There are two many dpendency graph component in between `tensorflow/core/platform:env` and `libtensorflow_framework.so`/`_pywrap_tensorflow_internal.so` in bazel. As a result this PR tries to address the issue by split out the `Env::Default()` implementation out and only link it with `libtensorflow_framework.so` to guarantee `libtensorflow_framework.so` is the only place to include `Env::Default()`. Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
…rnal_impl Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
So that //tensorflow/core:lib includes `Env:Default` Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
|
Thanks @mihaimaruseac for the help. I have made additional changes to the PR and |
mihaimaruseac
approved these changes
Oct 7, 2020
google-ml-butler
bot
added
kokoro:force-run
gbaned
added
ready to pull
copybara-service bot
pushed a commit
that referenced
this pull request
Oct 9, 2020
PiperOrigin-RevId: 336361140 Change-Id: I9c331eb08b71502c00235a9bad959372c82b46c1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
While working on modular file systems, noticed that tensorflow's pip package
includes two copies of
Env::Default()in two shared objects libraries.One is in
libtensorflow_framework.so, another is in_pywrap_tensorflow_internal.so.Because of that,
Env::Default()could return different instance depending on whena modular file system is called. The Lookup vs. Register discrepancy may confuse the file
system to believe the scheme (e.g.,
s3/gcs) is not registered (actually registeredby another
Env::Default()).The duplication of
Env::Default()is caused by the fact thattensorflow/core/platform:envdependency is included in both
libtensorflow_framework.soand_pywrap_tensorflow_internal.so.in bazel indirectly.
There are two many dpendency graph component in between
tensorflow/core/platform:envand
libtensorflow_framework.so/_pywrap_tensorflow_internal.soin bazel.As a result this PR tries to address the issue by split out the
Env::Default()implementationout and only link it with
libtensorflow_framework.soto guaranteelibtensorflow_framework.sois the only place to include
Env::Default().Signed-off-by: Yong Tang yong.tang.github@outlook.com