Use a shared library for shared code

[scoder]

Cython modules tend to be big (see #2102) and certain parts are the same for all modules. Especially in large installations, it would reduce the overall binary size to extract the shareable parts into a dedicated shared library that all modules link against.

Random notes:

• Some internal types are already shared across modules, but their code is still duplicated across all modules. The first imported Cython module (of a given Cython version) registers it globally for all others, which will then not use their own implementations.
• Cython can already extract its helpers into dedicated header files to reduce the build overhead. A shared library could be based on that.
• Much of the performance of Cython code comes from inlined (and compile time streamlined) functions. Extracting those into a shared library would probably slow things down visibly, but could be a tradeoff.
• Using a shared library adds quite some complexity. The library could be used globally for a Python installation (in which case it's unclear how to find it and which one to use if there are multiple), or local to an installed package. The latter is probably simpler but still requires some coordination on user side. A global library could come in as a separate package dependency that the Cython project could provide, but would introduce an exact version dependency based on the Cython version that a package was built with.
• It is not entirely clear how to deal with the runtime linking here. A real shared library would have to be loaded together with the Cython module, i.e. there would have to be a way for the shared library loader to find it. It might be easier (and more portable) to use a Python extension module and explicitly import the code from there via capsules, in the same way that Cython currently imports its shared types from the first imported module.
• Such a shared library would have to be complete in order to be usable by all Cython generated modules. It would thus be larger than what most single Cython modules (and even some small sets of Cython modules) actually need.
• The potential gain is unclear. For large installations, there is certainly a gain even if only the shared types are extracted from each of the modules. However, large installations are rare. If a shared library is created on a per-package basis, there will still be a lot of duplication in a given Python installation.

[jakirkham]

Thanks for writing this up @scoder. This is a very useful discussion to have.

It sounds like the first two points (internal types and helpers) are a good starting point for this shared library. Inlined functions would probably not be good in a shared library. That said, these should be profiled (if they haven't been already) to make sure inlining happens and is of significant value.

As to shared library management (points 4 and 5), the typical way, which I'm sure you know, to deal with this is use a SONAME. However the challenge becomes how do we install this. One option is we actually release a shared library with each Cython version to PyPI and anywhere else that include the SONAME in the package name (e.g. libcython-0.28). This makes it possible to install multiple versions and to ensure that code loads a compatible version. Similar strategies already occur in other package managers, which could easily adopt this. Managing this on PyPI may be a bit of a challenge without some improvements.

It's hard to reason about things like size without a simple prototype to inform one's intuition. Could you please suggest a few functions that might be reasonable for this sort of thing and/or where to look for them?

[robertwb]

Much of the utility code in https://github.com/cython/cython/tree/master/Cython/Utility would be ripe candidates for this sort of thing. Restricting the scope to a single (monolithically compiled) package would be simpler, and I'd guess the most benefit would be for those packages that have many, many cython modules in them anyways. (There's also been a related desire to compile multiple modules together in a single shared object library; I wonder if we did this if there are tools for deduplicating .so files that have multiple copies of the same exact function statically linked in... If there is such a tool I wonder if it could give a ballpark estimate for what potential savings there might be for a given collection of .so files.) We could also ship a cython_runtime package, with wheels, on PyPi and allow any projects to link against that (either via the C library linking level, or, simpler, via the same methods we use for cimport).

…

On Tue, Jun 19, 2018 at 4:29 PM, jakirkham ***@***.***> wrote: Thanks for writing this up @scoder <https://github.com/scoder>. This is a very useful discussion to have. It sounds like the first two points (internal types and helpers) are a good starting point for this shared library. Inlined functions would probably not be good in a shared library. That said, these should be profiled (if they haven't been already) to make sure inlining happens and is of significant value. As to shared library management (points 4 and 5), the typical way, which I'm sure you know, to deal with this is use a SONAME. However the challenge becomes how do we install this. One option is we actually release a shared library with each Cython version to PyPI and anywhere else that include the SONAME in the package name (e.g. libcython-0.28). This makes it possible to install multiple versions and to ensure that code loads a compatible version. Similar strategies already occur in other package managers, which could easily adopt this. Managing this on PyPI may be a bit of a challenge without some improvements. It's hard to reason about things like size without a simple prototype to inform one's intuition. Could you please suggest a few functions that might be reasonable for this sort of thing and/or where to look for them? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#2356 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAdqgSNwW4xKZFPA_QXhLpeDUcyQjGxMks5t-YlCgaJpZM4UqFUV> .

[jakirkham]

I wonder if we did this if there are tools for deduplicating .so files that have multiple copies of the same exact function statically linked in...

Good question. @mingwandroid, do you know of such a tool?

[scoder]

Here's an initial plan:

• The utility code generation is split into multiple phases to accommodate for C's prototype declarations, macros, implementation part, cleanup, etc. We'll introduce a new stage 'shared' that will simply drop its code right behind the impl by default, but marks utility code blocks that can be shared.
• Split the existing utility code impl sections into an impl part that needs to stay in each module and a shared part that can be reused across modules. Start with CythonFunction.c, Coroutine.c and AsyncGen.c, potentially also the memoryviews support, but that is probably more difficult.
• Add a new option shared_module to cythonize() that takes the qualified module name of a shared module to generate. For example, the lxml package would say something like shared_module="lxml._cython_shared".
• Make Cython generate the shared module on request, by dropping all the shared utility code into it, and also the code that exports them as the Cython ABI module (which already exists, see ImportExport.c and the related code snippets in the utility code files named above).
• Make Cython exclude the shared code from all the other modules and instead only use the existing shared importing code (which is used to share types and functions across Cython modules that currently duplicate their implementations).
• Don't forget to put a runtime version check into the sharing code to detect the case where only some of the modules in the package get rebuilt, but with a different Cython version. If the Cython version that generated the shared module does not match the version that generated the module that tries to use it, it should fail on import with a clear error message that the whole package needs to be recompiled with a single Cython version.

This is a bit of work, but seems relatively easy to do. Help is welcome.

[ncoghlan]

Noting some prior art in this space: PyQT's SIP support module is designed in such that it preserves its ABI, such that you don't need the exact same version of the support library, just a recent enough version that it provides all the required APIs.

Shared module on PyPI: https://pypi.org/project/SIP/

[da-woods]

Very crude estimate of the potential saving, just looking at a compiled version of Cython's visitor.py (defining CFLAGS="-DCYTHON_INLINE=" doesn't change all that much)

It compiled to ~2MB. strip takes it down to ~300kB.

Running:

nm --print-size --radix=d Cython/Compiler/Visitor.cpython-38-x86_64-linux-gnu.so | grep "__[pP][yY][xX]" | grep --invert-match "6Cython" | grep --invert-match "__pyx_[kn]" | grep --invert-match "string_tab" | grep --invert-match "pymod_exec_Visitor" > vistor_size.txt

That gets all the symbols starting with __pyx. But drops those starting with 6Cython (because they're user functions), plus some constants, plus the string tab, plus the module init function.

That leaves ~19kB of symbols. The majority of those are Cython "shared code" (but the filter isn't perfect). My view is that this is heading towards "not really worth it"

[robertwb]

Nice analysis. The true value likely isn't that far off, and the savings here pretty limited (especially compared to the complexity of making this work well). Also, though there are certainly some exceptions, Cython utility code tends towards those things that are either small enough to inline (vs. just calling the Python C API directly, or massaging parameters around a Python C API call) or specialized for the situation (e.g. the conversion operators).

Note that the sharing of types is so that we can do things like instance (or even equality) checks across modules.

I do think there may still be value in linking multiple modules into the same shared library, but the space savings do look fairly minimal.

[skaughtx0r]

Are you saying that the shared lib only has 19kb worth of shared symbols?

While that doesn't sound like a lot, in our particular case we're generating about 500+ separate cython .so files, so it does add up. I was under the impression that doing this could reduce the size more than that though, so I agree that it might not really be worth it.

[jakirkham]

Agreed I think this becomes more noticeable when considering shipping a library or even a large number of packages (containing various libraries)

[da-woods]

Are you saying that the shared lib only has 19kb worth of shared symbols?

That was essentially my conclusion. There's two caveats:

1. the example modules I looked at don't use memoryviews so they won't be included in my analysis. They do add a reasonably large block of C code
2. I don't know how debugging data is counted (i.e. are they associated with the functions? Or elsewhere?). It's possible I've failed to count the debugging information for the utility code and there might be a real saving from sharing that.

I think I've convinced myself that I won't be trying to implement this feature, but that doesn't mean that someone else shouldn't ;)

[navytux]

@da-woods, everyone, thanks for checking. I've also tried to repeat the analysis on one of my modules, and the outcome is directly the opposite: in my case the "shared part" takes ~ 60% of the total module size. My module do use memoryviews though. However, I believe memoryviews are so commonly used with Cython (ex scikit-learn), that their pesense should be assumed to be on by default. And that is exactly one of the reasons to keep Cython utility code in a shared library.

I might be wrong in my analysis somewhere, but I hope this observation demonstrates that solving this issue would have significant impact on code size, and also it would help to improve the speed of compilation (see also #3646 on this).

Please find details of my analisys below.

Thanks beforehand,
Kirill

---- 8< ----

Test module: mm.pyx
Size of compiled mm.so before strip: 445K
Size of mm.so after strip: 240K
Estimate for size of shared part: 140K

The estimate for shared part was built via:

$ nm --print-size --radix=d mm.so | grep "__[pP][yY][xX]" |  grep --invert-match "2mm" | grep --invert-match "__pyx_[kn]" | grep --invert-match "string_tab" >x.txt

$ awk '{s+=$2} END {print s}' <x.txt
144069

Top-100 size users from shared part consititute 112K of the 140K total:

$ sort --reverse -k2 x.txt  |head -100 |awk '{s+=$2} END {print s}'
115506

Those top size users can be seen to be indeed mostly related to memoryview and arrays functionality:

$ sort --reverse -k2 x.txt  |head -100
0000000000147049 0000000000005918 t __Pyx_InitCachedConstants
0000000000106502 0000000000005883 t __pyx_memview_slice
0000000000052264 0000000000005588 t __pyx_array___pyx_pf_15View_dot_MemoryView_5array___cinit__
0000000000137972 0000000000003847 t __pyx_format_from_typeinfo
0000000000083749 0000000000003668 t __pyx_memoryview_assign_item_from_object
0000000000080347 0000000000003402 t __pyx_memoryview_convert_item_to_object
0000000000117893 0000000000003275 t __pyx_memoryview_fromslice
0000000000128595 0000000000002844 t __pyx_memoryview_copy_contents
0000000000133527 0000000000002681 t __pyx_pf_15View_dot_MemoryView___pyx_unpickle_Enum
0000000000072766 0000000000002188 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_6__setitem__
0000000000065318 0000000000001852 t __pyx_pf___pyx_MemviewEnum___reduce_cython__
0000000000074954 0000000000001850 t __pyx_memoryview_is_slice
0000000000062709 0000000000001796 t __pyx_array_new
0000000000136208 0000000000001764 t __pyx_unpickle_Enum__set_state
0000000000113693 0000000000001683 t __pyx_pybuffer_index
0000000000050602 0000000000001662 t __pyx_array___cinit__
0000000000078336 0000000000001596 t __pyx_memoryview_setitem_slice_assign_scalar
0000000000196427 0000000000001590 t __pyx_memoryview_copy_new_contig
0000000000076804 0000000000001532 t __pyx_memoryview_setitem_slice_assignment
0000000000187799 0000000000001415 t __Pyx_BufFmt_ProcessTypeChunk
0000000000125968 0000000000001395 t __pyx_memoryview_err_dim
0000000000071328 0000000000001387 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_4__getitem__
0000000000189853 0000000000001371 t __Pyx_BufFmt_CheckString
0000000000198017 0000000000001364 t __Pyx_PyInt_As_int
0000000000112385 0000000000001308 t __pyx_memoryview_slice_memviewslice
0000000000127363 0000000000001232 t __pyx_memoryview_err
0000000000070061 0000000000001214 t __pyx_memoryview_get_item_pointer
0000000000057901 0000000000001180 t __pyx_array___pyx_pf_15View_dot_MemoryView_5array_2__getbuffer__
0000000000153567 0000000000001156 t __Pyx_modinit_type_init_code
0000000000087466 0000000000001138 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_8__getbuffer__
0000000000194153 0000000000001137 t __Pyx_PyObject_to_MemoryviewSlice_dc_unsigned_char__const__
0000000000195290 0000000000001137 t __Pyx_PyObject_to_MemoryviewSlice_dc_unsigned_char
0000000000180507 0000000000001131 t __Pyx_setup_reduce
0000000000091171 0000000000001048 t __pyx_pf_15View_dot_MemoryView_10memoryview_10suboffsets___get__
0000000000124931 0000000000001037 t __pyx_memoryview_err_extents
0000000000171722 0000000000001015 t __Pyx_PyUnicode_Equals
0000000000168548 0000000000001011 t __Pyx_ParseOptionalKeywords
0000000000067753 0000000000000998 t __pyx_memoryview___cinit__
0000000000094673 0000000000000991 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_12__repr__
0000000000090150 0000000000000979 t __pyx_pf_15View_dot_MemoryView_10memoryview_7strides___get__
0000000000132550 0000000000000977 t __pyx_pw_15View_dot_MemoryView_1__pyx_unpickle_Enum
0000000000193187 0000000000000966 t __Pyx_ValidateAndInit_memviewslice
0000000000164874 0000000000000962 t __Pyx_PyFunction_FastCallDict
0000000000098980 0000000000000943 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_22copy_fortran
0000000000097991 0000000000000943 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_20copy
0000000000093639 0000000000000894 t __pyx_pf_15View_dot_MemoryView_10memoryview_4size___get__
0000000000100696 0000000000000863 t __pyx_memoryview_new
0000000000095706 0000000000000833 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_14__str__
0000000000200908 0000000000000798 t __Pyx_PyInt_As_size_t
0000000000200110 0000000000000798 t __Pyx_PyInt_As_off_t
0000000000201817 0000000000000798 t __Pyx_PyInt_As_long
0000000000068751 0000000000000775 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview___cinit__
0000000000059632 0000000000000754 t __pyx_array_get_memview
0000000000146307 0000000000000742 t __Pyx_InitCachedBuiltins
0000000000089373 0000000000000735 t __pyx_pf_15View_dot_MemoryView_10memoryview_5shape___get__
0000000000175846 0000000000000735 t __Pyx__GetException
0000000000124199 0000000000000732 t __pyx_memoryview_copy_data_to_temp
0000000000199381 0000000000000729 t __Pyx_TypeInfoToFormat
0000000000191224 0000000000000723 t __pyx_typeinfo_cmp
0000000000092915 0000000000000682 t __pyx_pf_15View_dot_MemoryView_10memoryview_6nbytes___get__
0000000000097288 0000000000000657 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_18is_f_contig
0000000000096585 0000000000000657 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_16is_c_contig
0000000000183144 0000000000000639 t __pyx_insert_code_object
0000000000189214 0000000000000639 t __pyx_buffmt_parse_array
0000000000181638 0000000000000638 t __Pyx_ImportType
0000000000173209 0000000000000634 t __Pyx_GetItemInt_Fast
0000000000178319 0000000000000633 t __Pyx_PyInt_AddObjC
0000000000115376 0000000000000606 t __pyx_memslice_transpose
0000000000170201 0000000000000579 t __Pyx_Raise
0000000000171161 0000000000000561 t __Pyx_PyBytes_Equals
0000000000191947 0000000000000552 t __pyx_check_strides
0000000000166880 0000000000000540 t __Pyx_init_memviewslice
0000000000122364 0000000000000534 t __pyx_memoryview_copy_object_from_slice
0000000000176581 0000000000000532 t __Pyx_Import
0000000000088646 0000000000000528 t __pyx_pf_15View_dot_MemoryView_10memoryview_1T___get__
0000000000069553 0000000000000508 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_2__dealloc__
0000000000064505 0000000000000497 t __pyx_MemviewEnum___init__
0000000000060503 0000000000000492 t __pyx_array___pyx_pf_15View_dot_MemoryView_5array_8__getattr__
0000000000061048 0000000000000492 t __pyx_array___pyx_pf_15View_dot_MemoryView_5array_10__getitem__
0000000000183783 0000000000000478 t __Pyx_CreateCodeObjectForTraceback
0000000000116052 0000000000000474 t __pyx_memoryviewslice_convert_item_to_object
0000000000143879 0000000000000452 t __pyx_tp_dealloc_memoryview
0000000000121913 0000000000000451 t __pyx_memoryview_copy_object
0000000000192741 0000000000000446 t __pyx_verify_contig
0000000000144573 0000000000000442 t __pyx_tp_clear_memoryview
0000000000121168 0000000000000438 t __pyx_memoryview_get_slice_from_memoryview
0000000000067223 0000000000000436 t __pyx_pf___pyx_MemviewEnum_2__setstate_cython__
0000000000182276 0000000000000436 t __Pyx_CLineForTraceback
0000000000203860 0000000000000433 t __Pyx_PyIndex_AsSsize_t
0000000000116526 0000000000000431 t __pyx_memoryviewslice_assign_item_from_object
0000000000167885 0000000000000419 t __Pyx_XDEC_MEMVIEW
0000000000079932 0000000000000415 t __pyx_memoryview_setitem_indexed
0000000000164462 0000000000000412 t __Pyx_PyFunction_FastCallNoKw
0000000000143470 0000000000000409 t __pyx_tp_new_memoryview
0000000000203457 0000000000000403 t __Pyx_PyNumber_IntOrLong
0000000000240928 0000000000000400 d __pyx_getsets_memoryview
0000000000240288 0000000000000392 d __pyx_type___pyx_MemviewEnum
0000000000242144 0000000000000392 d __pyx_type___pyx_memoryviewslice
0000000000241536 0000000000000392 d __pyx_type___pyx_memoryview
0000000000239776 0000000000000392 d __pyx_type___pyx_array

[robertwb]

Memory view code is often specialized for type and dimension (and a couple of other properties), which may make it both less sharable and harder to share. But maybe there's still a significant savings.

…

On Thu, Apr 8, 2021, 1:35 AM Kirill Smelkov ***@***.***> wrote: @da-woods <https://github.com/da-woods>, everyone, thanks for checking. I've also tried to repeat the analysis on one of my modules, and the outcome is directly the opposite: in my case the *"shared part" takes ~ 60% of the total module size*. My module do use memoryviews though. However, I believe memoryviews are so commonly used with Cython (ex scikit-learn), that their pesense should be assumed to be on by default. And that is *exactly* one of the reasons to keep Cython utility code in a shared library. I might be wrong in my analysis somewhere, but I hope this observation demonstrates that solving this issue would have significant impact on code size, and also it would help to improve the speed of compilation (see also #3646 <#3646> on this). Please find details of my analisys below. Thanks beforehand, Kirill ---- 8< ---- Test module: mm.pyx <https://lab.nexedi.com/kirr/wendelin.core/blob/36766e1d/wcfs/internal/mm.pyx> Size of compiled mm.so before strip: 445K Size of mm.so after strip: 240K Estimate for size of shared part: 140K The estimate for shared part was built via: $ nm --print-size --radix=d mm.so | grep "__[pP][yY][xX]" | grep --invert-match "2mm" | grep --invert-match "__pyx_[kn]" | grep --invert-match "string_tab" >x.txt $ awk '{s+=$2} END {print s}' <x.txt144069 Top-100 size users from shared part consititute 112K of the 140K total: $ sort --reverse -k2 x.txt |head -100 |awk '{s+=$2} END {print s}'115506 Those top size users can be seen to be indeed mostly related to memoryview and arrays functionality: $ sort --reverse -k2 x.txt |head -1000000000000147049 0000000000005918 t __Pyx_InitCachedConstants0000000000106502 0000000000005883 t __pyx_memview_slice0000000000052264 0000000000005588 t __pyx_array___pyx_pf_15View_dot_MemoryView_5array___cinit__0000000000137972 0000000000003847 t __pyx_format_from_typeinfo0000000000083749 0000000000003668 t __pyx_memoryview_assign_item_from_object0000000000080347 0000000000003402 t __pyx_memoryview_convert_item_to_object0000000000117893 0000000000003275 t __pyx_memoryview_fromslice0000000000128595 0000000000002844 t __pyx_memoryview_copy_contents0000000000133527 0000000000002681 t __pyx_pf_15View_dot_MemoryView___pyx_unpickle_Enum0000000000072766 0000000000002188 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_6__setitem__0000000000065318 0000000000001852 t __pyx_pf___pyx_MemviewEnum___reduce_cython__0000000000074954 0000000000001850 t __pyx_memoryview_is_slice0000000000062709 0000000000001796 t __pyx_array_new0000000000136208 0000000000001764 t __pyx_unpickle_Enum__set_state0000000000113693 0000000000001683 t __pyx_pybuffer_index0000000000050602 0000000000001662 t __pyx_array___cinit__0000000000078336 0000000000001596 t __pyx_memoryview_setitem_slice_assign_scalar0000000000196427 0000000000001590 t __pyx_memoryview_copy_new_contig0000000000076804 0000000000001532 t __pyx_memoryview_setitem_slice_assignment0000000000187799 0000000000001415 t __Pyx_BufFmt_ProcessTypeChunk0000000000125968 0000000000001395 t __pyx_memoryview_err_dim0000000000071328 0000000000001387 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_4__getitem__0000000000189853 0000000000001371 t __Pyx_BufFmt_CheckString0000000000198017 0000000000001364 t __Pyx_PyInt_As_int0000000000112385 0000000000001308 t __pyx_memoryview_slice_memviewslice0000000000127363 0000000000001232 t __pyx_memoryview_err0000000000070061 0000000000001214 t __pyx_memoryview_get_item_pointer0000000000057901 0000000000001180 t __pyx_array___pyx_pf_15View_dot_MemoryView_5array_2__getbuffer__0000000000153567 0000000000001156 t __Pyx_modinit_type_init_code0000000000087466 0000000000001138 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_8__getbuffer__0000000000194153 0000000000001137 t __Pyx_PyObject_to_MemoryviewSlice_dc_unsigned_char__const__0000000000195290 0000000000001137 t __Pyx_PyObject_to_MemoryviewSlice_dc_unsigned_char0000000000180507 0000000000001131 t __Pyx_setup_reduce0000000000091171 0000000000001048 t __pyx_pf_15View_dot_MemoryView_10memoryview_10suboffsets___get__0000000000124931 0000000000001037 t __pyx_memoryview_err_extents0000000000171722 0000000000001015 t __Pyx_PyUnicode_Equals0000000000168548 0000000000001011 t __Pyx_ParseOptionalKeywords0000000000067753 0000000000000998 t __pyx_memoryview___cinit__0000000000094673 0000000000000991 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_12__repr__0000000000090150 0000000000000979 t __pyx_pf_15View_dot_MemoryView_10memoryview_7strides___get__0000000000132550 0000000000000977 t __pyx_pw_15View_dot_MemoryView_1__pyx_unpickle_Enum0000000000193187 0000000000000966 t __Pyx_ValidateAndInit_memviewslice0000000000164874 0000000000000962 t __Pyx_PyFunction_FastCallDict0000000000098980 0000000000000943 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_22copy_fortran0000000000097991 0000000000000943 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_20copy0000000000093639 0000000000000894 t __pyx_pf_15View_dot_MemoryView_10memoryview_4size___get__0000000000100696 0000000000000863 t __pyx_memoryview_new0000000000095706 0000000000000833 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_14__str__0000000000200908 0000000000000798 t __Pyx_PyInt_As_size_t0000000000200110 0000000000000798 t __Pyx_PyInt_As_off_t0000000000201817 0000000000000798 t __Pyx_PyInt_As_long0000000000068751 0000000000000775 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview___cinit__0000000000059632 0000000000000754 t __pyx_array_get_memview0000000000146307 0000000000000742 t __Pyx_InitCachedBuiltins0000000000089373 0000000000000735 t __pyx_pf_15View_dot_MemoryView_10memoryview_5shape___get__0000000000175846 0000000000000735 t __Pyx__GetException0000000000124199 0000000000000732 t __pyx_memoryview_copy_data_to_temp0000000000199381 0000000000000729 t __Pyx_TypeInfoToFormat0000000000191224 0000000000000723 t __pyx_typeinfo_cmp0000000000092915 0000000000000682 t __pyx_pf_15View_dot_MemoryView_10memoryview_6nbytes___get__0000000000097288 0000000000000657 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_18is_f_contig0000000000096585 0000000000000657 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_16is_c_contig0000000000183144 0000000000000639 t __pyx_insert_code_object0000000000189214 0000000000000639 t __pyx_buffmt_parse_array0000000000181638 0000000000000638 t __Pyx_ImportType0000000000173209 0000000000000634 t __Pyx_GetItemInt_Fast0000000000178319 0000000000000633 t __Pyx_PyInt_AddObjC0000000000115376 0000000000000606 t __pyx_memslice_transpose0000000000170201 0000000000000579 t __Pyx_Raise0000000000171161 0000000000000561 t __Pyx_PyBytes_Equals0000000000191947 0000000000000552 t __pyx_check_strides0000000000166880 0000000000000540 t __Pyx_init_memviewslice0000000000122364 0000000000000534 t __pyx_memoryview_copy_object_from_slice0000000000176581 0000000000000532 t __Pyx_Import0000000000088646 0000000000000528 t __pyx_pf_15View_dot_MemoryView_10memoryview_1T___get__0000000000069553 0000000000000508 t __pyx_memoryview___pyx_pf_15View_dot_MemoryView_10memoryview_2__dealloc__0000000000064505 0000000000000497 t __pyx_MemviewEnum___init__0000000000060503 0000000000000492 t __pyx_array___pyx_pf_15View_dot_MemoryView_5array_8__getattr__0000000000061048 0000000000000492 t __pyx_array___pyx_pf_15View_dot_MemoryView_5array_10__getitem__0000000000183783 0000000000000478 t __Pyx_CreateCodeObjectForTraceback0000000000116052 0000000000000474 t __pyx_memoryviewslice_convert_item_to_object0000000000143879 0000000000000452 t __pyx_tp_dealloc_memoryview0000000000121913 0000000000000451 t __pyx_memoryview_copy_object0000000000192741 0000000000000446 t __pyx_verify_contig0000000000144573 0000000000000442 t __pyx_tp_clear_memoryview0000000000121168 0000000000000438 t __pyx_memoryview_get_slice_from_memoryview0000000000067223 0000000000000436 t __pyx_pf___pyx_MemviewEnum_2__setstate_cython__0000000000182276 0000000000000436 t __Pyx_CLineForTraceback0000000000203860 0000000000000433 t __Pyx_PyIndex_AsSsize_t0000000000116526 0000000000000431 t __pyx_memoryviewslice_assign_item_from_object0000000000167885 0000000000000419 t __Pyx_XDEC_MEMVIEW0000000000079932 0000000000000415 t __pyx_memoryview_setitem_indexed0000000000164462 0000000000000412 t __Pyx_PyFunction_FastCallNoKw0000000000143470 0000000000000409 t __pyx_tp_new_memoryview0000000000203457 0000000000000403 t __Pyx_PyNumber_IntOrLong0000000000240928 0000000000000400 d __pyx_getsets_memoryview0000000000240288 0000000000000392 d __pyx_type___pyx_MemviewEnum0000000000242144 0000000000000392 d __pyx_type___pyx_memoryviewslice0000000000241536 0000000000000392 d __pyx_type___pyx_memoryview0000000000239776 0000000000000392 d __pyx_type___pyx_array — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2356 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AADWVANJRM23PBE3TMNWE63THVTDXANCNFSM4FFIKUKQ> .

[da-woods]

A significant chunk of it is "only" specialized by 4 parameters

cython/Cython/Compiler/MemoryView.py

Lines 815 to 820 in a1dd447

	context = {
	'memview_struct_name': memview_objstruct_cname,
	'max_dims': Options.buffer_max_dims,
	'memviewslice_name': memviewslice_cname,
	'memslice_init': PyrexTypes.MemoryViewSliceType.default_value,
	}

which are usually the same. So it may not be that bad.

It does look like maybe the only thing worth sharing, but possibly the most difficult to share.

[rgommers]

Here is the size of all extensions built with Cython in a SciPy package (v1.8.0). Note that this comes from an installed SciPy from conda-forge, which is properly stripped (wheels are the same; all of SciPy is ~90 MB on disk and we need to do some surgery):

$ find . -name "*.so" | xargs du -sch   # output filtered manually to only include Cython extensions:
336K    ./linalg/_decomp_update.cpython-39-x86_64-linux-gnu.so
264K    ./linalg/_solve_toeplitz.cpython-39-x86_64-linux-gnu.so
792K    ./linalg/cython_lapack.cpython-39-x86_64-linux-gnu.so
304K    ./linalg/cython_blas.cpython-39-x86_64-linux-gnu.so
44K     ./linalg/_matfuncs_sqrtm_triu.cpython-39-x86_64-linux-gnu.so
488K    ./linalg/_cythonized_array_utils.cpython-39-x86_64-linux-gnu.so
240K    ./fftpack/convolve.cpython-39-x86_64-linux-gnu.so
56K     ./_lib/messagestream.cpython-39-x86_64-linux-gnu.so
84K     ./_lib/_ccallback_c.cpython-39-x86_64-linux-gnu.so
32K     ./_lib/_test_deprecation_def.cpython-39-x86_64-linux-gnu.so
208K    ./spatial/_voronoi.cpython-39-x86_64-linux-gnu.so
924K    ./spatial/_ckdtree.cpython-39-x86_64-linux-gnu.so
1.1M    ./spatial/_qhull.cpython-39-x86_64-linux-gnu.so
720K    ./spatial/transform/_rotation.cpython-39-x86_64-linux-gnu.so
212K    ./spatial/_hausdorff.cpython-39-x86_64-linux-gnu.so
1.5M    ./stats/_unuran/unuran_wrapper.cpython-39-x86_64-linux-gnu.so
260K    ./stats/_qmc_cy.cpython-39-x86_64-linux-gnu.so
264K    ./stats/_biasedurn.cpython-39-x86_64-linux-gnu.so
468K    ./stats/_boost/beta_ufunc.cpython-39-x86_64-linux-gnu.so
416K    ./stats/_boost/nbinom_ufunc.cpython-39-x86_64-linux-gnu.so
260K    ./stats/_boost/hypergeom_ufunc.cpython-39-x86_64-linux-gnu.so
412K    ./stats/_boost/binom_ufunc.cpython-39-x86_64-linux-gnu.so
664K    ./stats/_stats.cpython-39-x86_64-linux-gnu.so
228K    ./stats/_sobol.cpython-39-x86_64-linux-gnu.so
272K    ./special/_ufuncs_cxx.cpython-39-x86_64-linux-gnu.so
212K    ./special/_test_round.cpython-39-x86_64-linux-gnu.so
44K     ./special/_comb.cpython-39-x86_64-linux-gnu.so
3.9M    ./special/cython_special.cpython-39-x86_64-linux-gnu.so
100K    ./special/_ellip_harm_2.cpython-39-x86_64-linux-gnu.so
3.1M    ./special/_ufuncs.cpython-39-x86_64-linux-gnu.so
56K     ./io/matlab/_mio_utils.cpython-39-x86_64-linux-gnu.so
128K    ./io/matlab/_streams.cpython-39-x86_64-linux-gnu.so
228K    ./io/matlab/_mio5_utils.cpython-39-x86_64-linux-gnu.so
328K    ./interpolate/_bspl.cpython-39-x86_64-linux-gnu.so
408K    ./interpolate/interpnd.cpython-39-x86_64-linux-gnu.so
408K    ./interpolate/_ppoly.cpython-39-x86_64-linux-gnu.so
36K     ./signal/_spectral.cpython-39-x86_64-linux-gnu.so
344K    ./signal/_upfirdn_apply.cpython-39-x86_64-linux-gnu.so
36K     ./signal/_max_len_seq_inner.cpython-39-x86_64-linux-gnu.so
120K    ./signal/_sigtools.cpython-39-x86_64-linux-gnu.so
268K    ./signal/_peak_finding_utils.cpython-39-x86_64-linux-gnu.so
268K    ./signal/_sosfilt.cpython-39-x86_64-linux-gnu.so
112K    ./cluster/_vq.cpython-39-x86_64-linux-gnu.so
312K    ./cluster/_optimal_leaf_ordering.cpython-39-x86_64-linux-gnu.so
408K    ./cluster/_hierarchy.cpython-39-x86_64-linux-gnu.so
188K    ./optimize/_cobyla.cpython-39-x86_64-linux-gnu.so
192K    ./optimize/_lsq/givens_elimination.cpython-39-x86_64-linux-gnu.so
88K     ./optimize/cython_optimize/_zeros.cpython-39-x86_64-linux-gnu.so
144K    ./optimize/_moduleTNC.cpython-39-x86_64-linux-gnu.so
324K    ./optimize/_bglu_dense.cpython-39-x86_64-linux-gnu.so
352K    ./optimize/_trlib/_trlib.cpython-39-x86_64-linux-gnu.so
2.1M    ./optimize/_highs/_highs_wrapper.cpython-39-x86_64-linux-gnu.so
40K     ./optimize/_highs/_highs_constants.cpython-39-x86_64-linux-gnu.so
52K     ./optimize/_group_columns.cpython-39-x86_64-linux-gnu.so
372K    ./ndimage/_ni_label.cpython-39-x86_64-linux-gnu.so
64K     ./ndimage/_cytest.cpython-39-x86_64-linux-gnu.so
628K    ./sparse/_csparsetools.cpython-39-x86_64-linux-gnu.so
308K    ./sparse/csgraph/_matching.cpython-39-x86_64-linux-gnu.so
304K    ./sparse/csgraph/_reordering.cpython-39-x86_64-linux-gnu.so
224K    ./sparse/csgraph/_min_spanning_tree.cpython-39-x86_64-linux-gnu.so
456K    ./sparse/csgraph/_shortest_path.cpython-39-x86_64-linux-gnu.so
168K    ./sparse/csgraph/_traversal.cpython-39-x86_64-linux-gnu.so
328K    ./sparse/csgraph/_flow.cpython-39-x86_64-linux-gnu.so
196K    ./sparse/csgraph/_tools.cpython-39-x86_64-linux-gnu.so

This adds up to 27.9 MB, which is a lot. It's of course less clear what could be saved there. There's 64 extensions, the highest estimate above is ~140 kb per extension ~= 8.4 MB.