Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tensorflow] Add C port #14392

Merged
merged 136 commits into from
Nov 25, 2020
Merged
Show file tree
Hide file tree
Changes from 122 commits
Commits
Show all changes
136 commits
Select commit Hold shift + click to select a range
0ab169d
fix build issues occurring from default bazel config being used becau…
Jul 23, 2020
543e19d
[tensorflow-cc] Update CONTROL and ci.baseline.txt
LilyWangL Jul 24, 2020
820c455
fix also applies to windows static build
Jul 24, 2020
9643548
fix debug and static builds as well as library naming for non-linux
Aug 11, 2020
4860a01
upgrade tensorflow from v1.14 to v2.3
Aug 15, 2020
dd6d89f
fix execute process macro
Aug 15, 2020
0d611be
extend overriden execute_process to more than one COMMAND as there ac…
Aug 17, 2020
d3121ce
Merge pull request #1 from microsoft/master
jgehw Aug 17, 2020
ee969d7
merge branch 'fix-execute-process-macro2' into 'fix-static-builds-and…
Aug 17, 2020
ad5ca5f
added another patch required for tensorflow v2.3, fixed path and work…
Aug 18, 2020
c743e49
Revert "incorporate changes from microsoft:master"
jgehw Aug 18, 2020
2571bd1
Merge pull request #2 from jgehw/revert-1-master
jgehw Aug 18, 2020
98ecc0c
Revert "Revert "incorporate changes from microsoft:master""
jgehw Aug 18, 2020
ab8fdf7
Merge branch 'master' of https://github.com/microsoft/vcpkg
Aug 18, 2020
36143ac
Merge branch 'master' of https://github.com/microsoft/vcpkg
Aug 20, 2020
ea07bb9
final fixes for static build + improving out messages
Aug 20, 2020
a12d22f
enabling linux and osx in CI to see if it works now
Aug 20, 2020
6c34a08
Merge branch 'master' into fix-static-builds-and-more
jgehw Aug 20, 2020
6e31cd6
simplified code, fixed version numbers, fixed generated include cmake…
Aug 21, 2020
afc7f1e
fix failing postbuild check on handcrafted empty dummy library by spr…
Aug 21, 2020
898bbbe
remove dead code commit by mistake again
Aug 21, 2020
2ca1f94
Merge branch 'fix-static-builds-and-more' of https://github.com/jgehw…
Aug 21, 2020
1866a82
Merge branch 'master' of https://github.com/microsoft/vcpkg
Aug 21, 2020
350b151
improvements from code review
Aug 21, 2020
630e890
cleaner fix for debug code
Aug 21, 2020
2c66c4b
find pip3 in PATH (PYTHON3_DIR apparently not valid for pip3)
Aug 22, 2020
40954e4
fix error in python helper script
Aug 22, 2020
9451d33
Merge branch 'master' of https://github.com/microsoft/vcpkg
Aug 22, 2020
d198bf1
fix wrong libname in postbuild script
Aug 24, 2020
f6acaeb
fix python detection + switch to python on msys2 (instead of embedded…
Aug 24, 2020
efd0c93
fix order of arguments
jgehw Aug 24, 2020
5c1f7ae
fix command (it may contain spaces such as C:\Program Files\...)
jgehw Aug 25, 2020
3b95e18
revert last commit (root cause for CI failures is something different…
jgehw Aug 25, 2020
bfffec6
fix regex comparision
jgehw Aug 25, 2020
c9d08e9
fix linebreaks in generated file
jgehw Aug 25, 2020
2de4912
merge branch 'fix-execute-process-macro2' into 'fix-static-builds-and…
jgehw Aug 25, 2020
cdfdc65
fix CRT linkage
jgehw Aug 25, 2020
c8c12b8
refactor implemenation to avoid as much code duplication as possible …
jgehw Aug 25, 2020
71cf605
Merge branch 'master' into fix-static-builds-and-more
jgehw Aug 26, 2020
9f67738
Merge branch 'fix-static-builds-and-more' of https://github.com/jgehw…
jgehw Aug 26, 2020
5d09b86
fix version numbers in helper scripts
jgehw Aug 26, 2020
6cb4e5b
enable work-around for Windows until bazel fix is available
jgehw Aug 26, 2020
5376c01
install missing python3-pip on linux
jgehw Aug 26, 2020
692c742
fix linux build by patching
jgehw Aug 28, 2020
fbbde32
Merge branch 'master' of https://github.com/microsoft/vcpkg
jgehw Aug 28, 2020
37c3528
merge branch 'master' into 'fix-static-builds-and-more'
jgehw Aug 28, 2020
1246969
apply timeout feature now available via merged master branch
jgehw Aug 28, 2020
c918214
correct linux build patch
jgehw Aug 28, 2020
e78a3df
improve debug build patches
jgehw Aug 28, 2020
9103128
improve linux patch
jgehw Aug 28, 2020
fc4404a
temporarily add debug to inspect what's going on on macOS CI
jgehw Aug 29, 2020
cc3823d
Merge branch 'master' of https://github.com/microsoft/vcpkg
jgehw Aug 29, 2020
74cc334
remove temporary debug code and fix static linking scripts for linux …
jgehw Aug 30, 2020
ee1082f
fix regex escaping
jgehw Aug 30, 2020
381066a
fix ambiguous match while grepping for the framework link command
jgehw Aug 30, 2020
006a21a
extend fix of ambiguous match while grepping for the framework link c…
jgehw Aug 31, 2020
f7ee6c6
Merge branch 'master' of https://github.com/microsoft/vcpkg
jgehw Aug 31, 2020
b667b99
Merge branch 'master' into fix-static-builds-and-more
jgehw Sep 1, 2020
37f79c6
Merge branch 'master' of https://github.com/microsoft/vcpkg
jgehw Sep 1, 2020
d77a47c
fix what merge of master broke
jgehw Sep 1, 2020
74c4fc8
Merge branch 'master' into fix-static-builds-and-more
jgehw Sep 1, 2020
f81922c
fix more what got broken by merging master
jgehw Sep 1, 2020
f0d1c89
Merge branch 'master' of https://github.com/microsoft/vcpkg into master
jgehw Sep 2, 2020
3d8e2f2
remove "unofficial" from filename
jgehw Sep 2, 2020
2051e9e
added switch do distinct classic and manifest mode when generating co…
jgehw Sep 2, 2020
1241b67
create symlinks for libraries without version number
jgehw Sep 3, 2020
1da9b92
fix linux postbuild script
jgehw Sep 3, 2020
3df3b9b
Merge branch 'master' of https://github.com/microsoft/vcpkg into master
jgehw Sep 3, 2020
900512c
Merge branch 'master' into fix-static-builds-and-more
jgehw Sep 3, 2020
fb609b5
temporarily disable code making problems
jgehw Sep 3, 2020
ec68741
add note for linking on Linux and macOS
jgehw Sep 4, 2020
84af4a0
forget to add README file in previous commit
jgehw Sep 4, 2020
7f599a2
add file forgotton in macro fixing patch
jgehw Sep 4, 2020
067323c
Merge branch 'master' of https://github.com/microsoft/vcpkg into master
jgehw Sep 8, 2020
b8e9523
Merge branch 'master' of https://github.com/microsoft/vcpkg into master
jgehw Sep 10, 2020
bcda036
Merge branch 'master' into fix-static-builds-and-more
jgehw Sep 10, 2020
e4f15a8
fix python library path
jgehw Sep 11, 2020
da3cb5a
fix macOS static link command
jgehw Sep 11, 2020
7273f86
update linkage instructions in README
jgehw Sep 11, 2020
fdc5cf6
Update ports/tensorflow-cc/CONTROL
JackBoosY Sep 14, 2020
05be090
Update ports/tensorflow-cc/portfile.cmake
JackBoosY Sep 14, 2020
faf3447
Update scripts/ci.baseline.txt
JackBoosY Sep 14, 2020
3787012
use vcpkg_execute_required_process
jgehw Sep 14, 2020
0877229
pass C_FLAGS and CXX_FLAGS to bazel
jgehw Sep 14, 2020
eaf1d7c
fix INTERFACE_INCLUDE_DIRECTORIES
jgehw Sep 14, 2020
0a986ac
fix optional c/cxx arguments
jgehw Sep 17, 2020
bb63d8c
also add linker opts
jgehw Sep 22, 2020
79076a2
Merge branch 'pass-cflags-to-bazel' into fix-static-builds-and-more
jgehw Sep 30, 2020
ca81fb9
update README
jgehw Sep 30, 2020
8e16503
merge static libs into one
Oct 1, 2020
8677bdb
update README
Oct 1, 2020
39312c7
Merge branch 'master' of https://github.com/microsoft/vcpkg
Oct 1, 2020
79a26c8
Merge branch 'master' into fix-static-builds-and-more
Oct 1, 2020
6384898
Merge branch 'master' of https://github.com/microsoft/vcpkg into master
jgehw Oct 9, 2020
533895b
Merge branch 'master' into fix-static-builds-and-more
jgehw Oct 9, 2020
2ad839b
Merge branch 'master' of https://github.com/microsoft/vcpkg into master
jgehw Oct 15, 2020
f4e64da
Merge branch 'master' of https://github.com/microsoft/vcpkg into master
jgehw Oct 21, 2020
180925f
quote python path (it might contain spaces that don't get escaped ins…
jgehw Oct 22, 2020
4584d3a
fix python path also for static build
jgehw Oct 24, 2020
49f6f38
add arm(64) as currently unsupported arch
jgehw Oct 24, 2020
a174637
bazel 3.7 is available -> remove workaround
jgehw Oct 24, 2020
292013f
update README, remove necessary c-ares from deps
jgehw Oct 24, 2020
f8e004c
update msys package
Oct 24, 2020
a259f33
Merge branch 'fix-static-builds-and-more' of https://github.com/jgehw…
Oct 24, 2020
0ab5c4e
add uwp specific options, and minor general improvements
Oct 25, 2020
920af34
fix string replace
Oct 25, 2020
6594588
fix control file and windows path separator
Oct 25, 2020
0a22920
revert backslashes-fix -- the root cause was missing .exe extension
Oct 25, 2020
4ef9252
upgrade to tf 2.3.1
Oct 25, 2020
41e2f80
fix hard-coded version
Oct 25, 2020
7fa2689
Merge branch 'master' into fix-static-builds-and-more
jgehw Oct 26, 2020
fe1d24c
Merge branch 'master' of https://github.com/microsoft/vcpkg into master
jgehw Oct 27, 2020
ff1d1c9
Merge branch 'master' into fix-static-builds-and-more
jgehw Oct 27, 2020
f064862
remove uwp work-in-progress code so that PR can be merged
jgehw Oct 27, 2020
a68ae07
add [tensorflow] C API port
jgehw Oct 27, 2020
b3a6018
missing in previous commit
jgehw Oct 27, 2020
e2b95d1
fix include file
jgehw Oct 27, 2020
83149c4
remove unnecessary suffix
jgehw Oct 28, 2020
4840250
update README and print out usage info in portfile
jgehw Nov 4, 2020
24831cc
Merge branch 'fix-static-builds-and-more' into add-tensorflow-c
jgehw Nov 4, 2020
3898892
Merge branch 'master' of https://github.com/microsoft/vcpkg into master
jgehw Nov 12, 2020
06f7a9c
Merge branch 'master' into add-tensorflow-c
jgehw Nov 12, 2020
e1c55ed
Update ports/tensorflow-cc/CONTROL
jgehw Nov 14, 2020
51cae0d
Update ports/tensorflow/portfile.cmake
jgehw Nov 14, 2020
52a905d
follow-up to code review
jgehw Nov 14, 2020
d954930
fix suffix parameter
jgehw Nov 15, 2020
f3351f5
fix quoting
jgehw Nov 16, 2020
77559a6
Merge branch 'master' of https://github.com/microsoft/vcpkg into master
jgehw Nov 16, 2020
26cbca8
Merge branch 'master' into add-tensorflow-c
jgehw Nov 16, 2020
43039bc
extend linux patch
jgehw Nov 16, 2020
d005d4d
another try to fix quoting of possibly empty string parameter
jgehw Nov 16, 2020
6374836
different approach to fix empty string arguments
jgehw Nov 17, 2020
e45fdcf
update list of headers for tensorflow 2.3.1 (was still 2.3.0)
jgehw Nov 17, 2020
3b3ef9f
Update ports/tensorflow/CONTROL
jgehw Nov 18, 2020
bb1a320
move common stuff into tensorflow-common as discussed in review
jgehw Nov 20, 2020
d468baf
Apply suggestions from code review
jgehw Nov 21, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion ports/tensorflow-cc/CONTROL
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
Source: tensorflow-cc
Version: 2.3.1
jgehw marked this conversation as resolved.
Show resolved Hide resolved
Description: Library for computation using data flow graphs for scalable machine learning
Description: Library for computation using data flow graphs for scalable machine learning (C++ API version)
Supports: !(x86|arm|uwp)
5 changes: 3 additions & 2 deletions ports/tensorflow-cc/convert_lib_params_linux.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
import sys

version = sys.argv[1]
lib_suffix = sys.argv[2]

with open(f"libtensorflow_cc.a.{version}-2.params", "w") as f_out:
with open(f"libtensorflow{lib_suffix}.a.{version}-2.params", "w") as f_out:
parts = []
with open(f"libtensorflow_framework.so.{version}-2.params", "r") as f_in:
skip_next = False
Expand All @@ -16,7 +17,7 @@
f_out.write(line)
parts.append(line)
parts = set(parts)
with open(f"libtensorflow_cc.so.{version}-2.params", "r") as f_in:
with open(f"libtensorflow{lib_suffix}.so.{version}-2.params", "r") as f_in:
skip_next = False
for line in f_in:
if skip_next:
Expand Down
9 changes: 5 additions & 4 deletions ports/tensorflow-cc/convert_lib_params_windows.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,13 @@
import os
import sys

lib_suffix = sys.argv[1]

file_no = 1
with open("tensorflow_cc.dll-2.params", "r") as f_in:
with open(f"tensorflow{lib_suffix}.dll-2.params", "r") as f_in:
lib_name = None
acc_size = 0
f_out = open("tensorflow_cc.lib-2.params-part1", "w")
f_out = open(f"tensorflow{lib_suffix}.lib-2.params-part1", "w")
for line in f_in:
if line.startswith("/OUT:"):
lib_name = line
Expand All @@ -18,10 +20,9 @@
# we need to split the library if it is >4GB, because it's not supported even on x64 Windows
f_out.close()
file_no += 1
f_out = open(f"tensorflow_cc.lib-2.params-part{file_no}", "w")
f_out = open(f"tensorflow{lib_suffix}.lib-2.params-part{file_no}", "w")
acc_size = 0
f_out.write(lib_name.replace(".dll", f"-part{file_no}.lib"))
acc_size += size
f_out.write(line)
f_out.close()

3 changes: 2 additions & 1 deletion ports/tensorflow-cc/generate_static_link_cmd_linux.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
import re
import sys

lib_suffix = sys.argv[2]
with open(sys.argv[1], "r") as f_in:
with open("static_link.sh", "w") as f_out:
p_cd = re.compile("^\\((cd .*) && \\\\$")
p_linker = re.compile("^\\s*(.+)gcc.+(@bazel-out\\S+libtensorflow_cc\\.so\\.\\d\\.\\d\\.\\d-2\\.params).*")
p_linker = re.compile("^\\s*(.+)gcc.+(@bazel-out\\S+libtensorflow" + lib_suffix + "\\.so\\.\\d\\.\\d\\.\\d-2\\.params).*")
f_out.write("#!/bin/bash\n# note: ar/binutils version 2.27 required to support output files > 4GB\n")
env = []
for line in f_in:
Expand Down
5 changes: 3 additions & 2 deletions ports/tensorflow-cc/generate_static_link_cmd_macos.py
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
import re
import sys

lib_suffix = sys.argv[2]
with open(sys.argv[1], "r") as f_in:
with open("static_link.sh", "w") as f_out:
p_cd = re.compile("^\\((cd .*) && \\\\$")
p_linker1 = re.compile("^.*cc_wrapper.sh.+-shared.+-o (bazel-out\\S+libtensorflow_cc\\.\\d\\.\\d\\.\\d\\.dylib)")
p_linker1 = re.compile("^.*cc_wrapper.sh.+-shared.+-o (bazel-out\\S+libtensorflow" + lib_suffix + "\\.\\d\\.\\d\\.\\d\\.dylib)")
p_linker2 = re.compile("^.*cc_wrapper.sh.+-shared.+-o (bazel-out\\S+libtensorflow_framework\\.\\d\\.\\d\\.\\d\\.dylib)")
f_out.write("#!/bin/bash\n# note: ar/binutils version 2.27 required to support output files > 4GB\n")
env = []
Expand Down Expand Up @@ -39,7 +40,7 @@
tmp = [t[16:] for t in tokens if t.startswith("-Wl,-force_load,")]
old = set(parts)
parts += [t for t in tmp if t not in old]
line = f"libtool -static -o {m2.group(1).replace('_framework', '_cc').replace('.dylib', '.a')} {' '.join(parts)}\n"
line = f"libtool -static -o {m2.group(1).replace('_framework', lib_suffix).replace('.dylib', '.a')} {' '.join(parts)}\n"
f_out.write(line)
break
else:
Expand Down
5 changes: 3 additions & 2 deletions ports/tensorflow-cc/generate_static_link_cmd_windows.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,11 @@
import re
import sys

lib_suffix = sys.argv[2]
with open(sys.argv[1], "r") as f_in:
with open("static_link.bat", "w") as f_out:
p_setenv = re.compile("^\s*(SET .+=.*)$")
p_linker = re.compile(".+link\\.exe.+tensorflow_cc\\.dll-2\\.params.*")
p_linker = re.compile(".+link\\.exe.+tensorflow" + lib_suffix + "\\.dll-2\\.params.*")
env = []
for line in f_in:
if line.startswith("cd"):
Expand All @@ -28,7 +29,7 @@
t = t[:-len("link.exe")] + "lib.exe\""
elif t == "/DLL" or t.lower()[1:].startswith("defaultlib:") or t.lower()[1:].startswith("ignore") or t.startswith("/OPT:") or t.startswith("/DEF:") or t.startswith("/DEBUG:") or t.startswith("/INCREMENTAL:"):
continue
elif t[0] == '@' and t.endswith("tensorflow_cc.dll-2.params"):
elif t[0] == '@' and t.endswith(f"tensorflow{lib_suffix}.dll-2.params"):
t = t[:-len("dll-2.params")] + "lib-2.params-part1"
params_file = t[1:-len("-part1")]
line += t + " "
Expand Down