Increasing package size limit for apache-flink

[sunjincheng121]

Hi, I would like to request you to increase the size limit of the package "apache-flink" to 400MB, because in the next release, apache-flink will package the Flink Cord JARs files and opt(connector's JRAs), in that case the size will be increased almost 400MB. Please help me increase the upload limit, thank you. There is a developmental release in both PyPI and Test PyPI for the moment, but it is incomplete. It's actual just a test release package.

A link to your project on PyPI (or Test PyPI):

https://test.pypi.org/project/apache-flink/
https://pypi.org/project/apache-flink/

The size of your release, in megabytes:
Almost 400MB (After official release）

Which index/indexes you need the increase for (PyPI, Test PyPI, or both)：
Both

A brief description of your project, including the reason for the additional size.
Apache-Flink allows you to write Flink programs in Python. The additional size comes from the flink JARs files.

BTW：The discussion and vote mail thread can be found here: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Publish-the-PyFlink-into-PyPI-td31201.html

[sunjincheng121]

I appreciate if you can have look at this apply @jamadden @nlhkabu
Best, Jincheng

[jamadden]

There is a previous distribution of apache flink on PyPI. In #6116 it was given a limit of 300MB, though it appears that was never used. If this project is to officially replace that one, perhaps there could be some coordination to remove it from PyPI so as to prevent user confusion and excess resource usage.

300MB is on the very upper end of sizes that moderators are suggested to approve without consulting with other moderators. 400MB is in the area that requires more scrutiny. In particular, if "the package is attempting to ship ... non-Python stuff like OpenJDK" moderators are suggested to seek further clarification from the requestor and discuss with the other moderators.

Besides the PyPi resource usage, large packages don't produce a very good user experience, so we like to help maintainers find alternate solutions.

@sunjincheng121 could you please provide more details about why this project needs to be so large? Were any other distribution options considered and rejected (and if so, why)? For example, one suggestion is that the package could "side-load large models or datasets via downloads from elsewhere at runtime/installtime." Or, since you mention multiple different JARs, another option might be to package them independently and "break the package into smaller packages that depend on each other as necessary."

[sunjincheng121]

Hi @jamadden, I am very sorry that I did not explain the relationship between https://pypi.org/project/pyflink/ and https://pypi.org/project/apache-flink/.
Before we did the Flink community discussion[1], we wanted to use pyflink as the release project, but pyflink was created by other people not related to Apache Flink. After discussions in the Flink community, we finally decided to create a new apach-flink project as the official release.(The project(pyflink) is not part of Flink, we can not delete it for now.)

Regarding the size of the package, I am worried about the future expansion, so I applied for 400MB. It is also possible to apply 300MB at present. Other packages if needed in the future, such as your suggestions, we can also use the script to download dynamically.

So a brief summary:

1. https://test.pypi.org/project/apache-flink/ is an official project of Apache Flink (on behalf of the Flink PMC)
2. Apply for package size up to 300MB

If there is anything I do not explanation clearly, please let me know :)

Best, Jincheng

[1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Publish-the-PyFlink-into-PyPI-td31201.html

[sunjincheng121]

@jamadden @nlhkabu is there something I should explain? :)

[dianfu]

@jamadden @sunjincheng121 Thanks a lot for the discussion. Any progress? @jamadden Do you think it makes sense to increase the package size for now and we can continue to investigate other options such as breaking the package into a few smaller packages per your suggestion.

[sunjincheng121]

@jamadden @nlhkabu @di could you please have look at this application, Is there any suggestions for me to explain or improve?

[jamadden]

My apologies for the delay, my time for this has been extremely limited lately.

Thank you for the detailed information. Based on it, I've taken the following actions:

• Increase apache-flink to 300MB on both pypi.org and test.pypi.org (larger sizes might need other moderators to take a look)
• Restore pyflink to the default limit, reverting Increasing package size limit for pyflink #6116

[sunjincheng121]

Thanks @jamadden !