Accidental drop of Python 3.6 support? Colab incompatibility due to hard-pinned protobuf version.

[bluenote10]

This issue refers to the Python package of tensorflowjs.

I was trying to use Google Magenta's ddsp on Colab, but ran into a major issue trying to installing it. A simple pip install ddsp has become super slow (half an hour or so). Since this is a mandatory step in every Colab session this should ideally be seconds, not in the order of hours.

Investigating the cause of the issue revealed that ddsp depends on tensorflowjs. The latest version of tensorflowjs has a hard-pinned requirement on protobuf==3.20.0. Of course, hard pinned requirements are never great when a package is supposed to be used as a library (like how ddsp uses it). Even worse: The version 3.20.0 of protobuf seems to be the first version that has dropped Python 3.6 support. Since Google Colab enforces Python 3.6 that hard-pinned version simply cannot installed. Under Python 3.6 you'll get:

$ pip install protobuf==3.20.0                                                                                                                                                                                                                                          
ERROR: Could not find a version that satisfies the requirement protobuf==3.20.0 (from versions: 2.0.0b0, 2.0.3, 2.3.0, 2.4.1, 2.5.0, 2.6.0, 2.6.1, 3.0.0a2, 3.0.0a3, 3.0.0b1, 3.0.0b1.post1, 3.0.0b1.post2, 3.0.0b2, 3.0.0b2.post1, 3.0.0b2.post2, 3.0.0b3, 3.0.0b4, 3.0.0, 3.1.0, 3.1.0.post1, 3.2.0rc1, 3.2.0rc1.post1, 3.2.0rc2, 3.2.0, 3.3.0, 3.4.0, 3.5.0.post1, 3.5.1, 3.5.2, 3.5.2.post1, 3.6.0, 3.6.1, 3.7.0rc2, 3.7.0rc3, 3.7.0, 3.7.1, 3.8.0rc1, 3.8.0, 3.9.0rc1, 3.9.0, 3.9.1, 3.9.2, 3.10.0rc1, 3.10.0, 3.11.0rc1, 3.11.0rc2, 3.11.0, 3.11.1, 3.11.2, 3.11.3, 3.12.0rc1, 3.12.0rc2, 3.12.0, 3.12.1, 3.12.2, 3.12.4, 3.13.0rc3, 3.13.0, 3.14.0rc1, 3.14.0rc2, 3.14.0rc3, 3.14.0, 3.15.0rc1, 3.15.0rc2, 3.15.0, 3.15.1, 3.15.2, 3.15.3, 3.15.4, 3.15.5, 3.15.6, 3.15.7, 3.15.8, 3.16.0rc1, 3.16.0rc2, 3.16.0, 3.17.0rc1, 3.17.0rc2, 3.17.0, 3.17.1, 3.17.2, 3.17.3, 3.18.0rc1, 3.18.0rc2, 3.18.0, 3.18.1, 3.19.0rc1, 3.19.0rc2, 3.19.0, 3.19.1, 3.19.2, 3.19.3, 3.19.4, 4.0.0rc1, 4.0.0rc2, 4.21.0rc1, 4.21.0rc2, 4.21.0)
ERROR: No matching distribution found for protobuf==3.20.0

This means that pip will go into its dependency resolution backtracking mode resulting in the infamous "This is taking longer than usual..." warnings. Apparently it downloads basically every single version of tensorflow related packages to figure out a solution, which of course makes it super slow.

For now I'm opening this issue to create awareness of the issue and discuss possible solutions. Dropping Python 3.6 support will cause friction within the ecosystem due to Colab depending on it. There are probably two solutions:

• Find a way to still support Python 3.6 in tensorflowjs.
• Accepting dropping support of Python 3.6 in tensorflowjs. In this case it may help to mark the latest release of tensorflowjs explicitly as "non Python 3.6" so that pip doesn't even try to pick it up and only finds the Python 3.6 indirectly via the protobuf pinning. And in any case, the == should be avoided in favor of >=. For instance, ddsp also depends on a package called note-seq. This dependency in turn has a dependency on protobuf>=4.21.2 which of course can never fit to the other transitive dependency of tensorflowjs requiring protobuf==3.20.0.

[pyu10055]

@bluenote10 I noticed python default version is 3.7.13 instead of 3.6, is that correct?

[bluenote10]

@pyu10055 Up until recently it still was Python 3.6, but indeed it looks like it has been updated to 3.7 which is great news!

Nonetheless there are still issues caused by the exact pinning of protobuf==3.20.0 on the tensorflowjs side. For instance, it causes an internal dependency conflict on protobuf for ddsp and it also seems to conflict with the default Colab environment: