Prepare Upgrade to TensorFlow 2

[lastzero]

As a developer, I want to use the latest TensorFlow version so that I continue to get updates and can use all features.

We are currently using TensorFlow 1.14.0

Acceptance Criteria:

• Identify all major steps necessary to upgrade our Docker image and the application code to TensorFlow 2 (e.g. check documentation of TensorFlow Go library)
• Note that we also support arm64, not just x86, so we need to be able to build TF2 for/on the Raspberry Pi 3/4 (64bit Linux)
• Test our existing models with TensorFlow 2 and make sure they continue to work as expected (run existing unit tests and do manual acceptance tests)
• Document your findings here in this ticket plus add a summary to our wiki (stream of issue comments can be confusing to use as todo list to perform the actual upgrade later)

[guysoft]

I can help with the Rpi build.
You could also use qemu to build, but it might take days, so I would recommend just using a Pi.

[lastzero]

Good to know qemu is so slow! So not an option for us in practice.

We have a Raspberry Pi 4 with 4 GB RAM and an Nvidia Jetson in our lab for development.

[guysoft]

Good to know qemu is so slow! So not an option for us in practice.

Its usually OK, but tensorflow took 7 hours in Raspberrypi 4 running on two threads. If you have a super strong intel machine it might be worth it, but I'd recommend using a Pi just for now, since its a once-per-version build.

[lastzero]

We're also building the photoprism-arm64 image on the Raspberry Pi 4 using your 64bit Ubuntu image at the moment. It's reasonable fast, so the question is if it's worth finding a different (cloud based) solution.

Edit: Would qemu profit from more cores? We think about getting an AMD Threadripper...

[guysoft]

You could also use multi-arch to build it using docker hub. If you use a local x86_64 machine it might also build.

[savegame]

i built your project in FreeNAS Jail (FreeBSD 11.2 at this moment) , but it has only Tensorflow 1.14 :) it would be sad if i cant build tensorflow 2 in freebsd ... is it possible make it compatible with tensorflow v1 and v2 in same time?

[lastzero]

Where do you have TF 1.14 from? We'll probably stick with it until after our first stable release.

[savegame]

i build TF 1.14 from FreeBSD ports

[lastzero]

i build TF 1.14 from FreeBSD ports

Can you check for us if / when they have TF2? I've used FreeBSD many years ago. You're also welcome to write a howto for our docs.

[savegame]

i'll try later write some howto about FreeBSD (especially FreeNAS jail) (dont have much time 😞 )

[McQueen2063]

@savegame Sorry for being a bit off topic in this thread, but I try to build photoprism under FreeBSD as well (that is, a FreeBSD 12.1 jail, not inside FreeNAS, but this shouldn't matter).
I did install tensorflow from pkg (py37-tensorflow-1.14.0_3), but gmake all later fails with:

go build -v ./...
github.com/tensorflow/tensorflow/tensorflow/go
# github.com/tensorflow/tensorflow/tensorflow/go
ld: error: unable to find library -ltensorflow
clang: error: linker command failed with exit code 1 (use -v to see invocation)
gmake: *** [Makefile:59: dep-go] Error 2

Well, yes, there is no libtensorflow.so.
How did you proceed from there?

[lastzero]

You might need to build it from source. Takes a couple of hours to compile though.

[McQueen2063]

I'm building /usr/ports/science/py-tensorflow right now. I wonder why the outcome of building the port is different than the pkg, but I'll give it a try anyways :)

[lastzero]

You need TensorFlow for C, not Python.

[lastzero]

As of right now, TensorFlow for Go seems broken for the average developer:

tensorflow/tensorflow#41808

Their downloadable C library is also outdated. CPU requirements not really documented as well.

[guysoft]

Note there is a workaround: tensorflow/tensorflow#41808 (comment)
And it does work in master: tensorflow/tensorflow#41808 (comment)

[lastzero]

Yeah, but we really don't have time for hacks and workarounds right now. Need to compile it for different CPU architectures as well, including GPU support. Would be OK doing this as a full time job if there was nothing else to do.

[gyto6]

In case you need it, I think this link might help to prepare TF2 (maybe 3) support for Nvidia GPU.
https://pixinsight.com/forum/index.php?threads/gpu-accelerated-starnet-v2-working-with-cuda-and-libtensorflow-gpu-under-linux.18180/

[lastzero]

Since Google / the TensorFlow team decided to make it harder to use TensorFlow with Go (wonder why...), the next best/only option is to migrate our backend to this fork maintained by @galeone:

• https://github.com/galeone/tensorflow

Ideally, we do this when the fork is getting updated to the then latest version in the upcoming weeks/months. Waiting is fine if it saves us from having to upgrade to a new TF version twice in short order.

In our particular situation, upgrading also means that we (may) have to build and distribute new C libraries for all supported architectures and operating systems. This can be a lot of extra work, unless there are existing binaries, which has often not been the case in the past. We therefore plan accordingly, as hoping for luck is not a strategy.

[galeone]

Eventually, I updated the fork today.

You can depend on https://github.com/galeone/tensorflow/tree/r2.9-go

[lastzero]

Wow, that was quick! Thanks a lot! So now it's up to us to update our dependencies and refactor existing code as needed 🥳

Do you happen to know if binary models exported with TF 1.x remain compatible or how to convert them if needed?

[galeone]

In TensorFlow 2.x you can only use the SavedModel serialization format. Hence, if you have some "frozen" models you can't import and use them.

[galeone]

FYI I updated https://github.com/galeone/tensorflow to TensorFlow 2.10 (https://github.com/galeone/tensorflow/tree/r2.10-go)

You can update via go get -u github.com/galeone/tensorflow/tensorflow/go@r2.10-go

Moreover, if you use (or plan to use) galeone/tfgo the dependency of tfgo on galeone/tensorflow is already updated.

[lastzero]

@galeone Thanks for the update! I really appreciate it! Due to limited resources, we unfortunately have to wait with the upgrade until we have released the extended multi-user support.

[lapo-luchini]

Can you check for us if / when they have TF2? I've used FreeBSD many years ago. You're also welcome to write a howto for our docs.

FreeBSD Ports seems to have tensorflow 2.9.1 but tensorflox 1.x never made in officially (though I have a personal binary pkg for that for personal use published on GitHub) because porting bazel-based stuff seems to be quite complex and in the end no one finished it properly for an old version, so I look forward to when 2.x will be supported too. 😁

[lastzero]

@lapo-luchini Thank you for letting us know! We appreciate any help we can get. However, I understand that migrating to TF2 might be too much of a challenge for contributors. I'll otherwise give it a try once we've finished adding support for OAuth2, OIDC, and 2FA (see e.g. #3943) and have resolved the known issues related to facial recognition.