Discussing possible fusion with pypdfium2-team development fork

[mara004]

As you might know already, I've been working on an overhaul of ctypesgen at
https://github.com/pypdfium2-team/ctypesgen/tree/pypdfium2

Here are some resources to get an overview of the design direction I've been going:¹
https://github.com/pypdfium2-team/ctypesgen/blob/pypdfium2/README.md
https://github.com/pypdfium2-team/ctypesgen/blob/pypdfium2/docs/CHANGELOG.md#unreleased-v2
pypdfium2-team#1
master...pypdfium2-team:ctypesgen:pypdfium2

I believe the changes contain significant improvements/modernizations from both ctypesgen dev and embedder perspective, including logic corrections / bug fixes (e.g. to common headers / module linking / external preamble), and new features (e.g. symbol filters, style options).
Also I've modernized the test suite, started pathlib and f-string migration, and cleaned up many code passages (e.g. python printer, operations, library loader, preamble).
Particular goals are to value the Keep It Simple principle, relieve ctypesgen of most custom wrappers, stick to plain ctypes as much as possible, and avoid code smell.

If there's a way to find a common ground, it would be great if this development work could eventually (not quite yet) be merged back here to share the same base and avoid fragmentation. Note however that it would not be possible or reasonable to do so without significant backward incompatibility, which is tied to the overhaul, though we could talk about adding a few compat-oriented extensions (e.g. a string helper such as pypdfium2-team#1 (comment)) as long as there's a lean, pollution free way to integrate.

I know I might not be able to actively work on ctypesgen in the somewhat near future, yet I believe this were the more suitable base for future development, and would be glad if the work were not lost or done twice in separate places.
To any future developers, please consider looking through these changes first before starting on the "old" codebase independently.

I just thought I'd open this issue as a sort of bulletin board for discussion, although my availability to reply may be limited.

---

For some reference, here's also a selection of issues and PRs I committed to:

Feedback / info / ideas

• Recognize structs from common header files in different wrapper modules #86 (comment)
• Feature request: option to treat certain pointers as opaque #20 (comment)
• Include statements included in target header files #87 (comment)
• With Python 3, seeing error LP_xyz instance instead of LP_xyz #92 (comment)
• Specify correct runtime (versus compile time) libraries on Linux #84 (comment)
• Name collision in ctypegen? #146 (comment)
• Build path stripping does not work correctly for header #160
• Mention that ctypesgen requires gcc #165 (comment)
• Correct definition of __slots__ on structs (must be defined in class body) #183 (comment)
• Provide CFUNCTYPE prototype name for function pointers defined in Structure #13 (comment)
• Improve conflicting names resolver #200
• Avoid overly defensive programming patterns #202
• Issue when binding to Python API (struct__typeobject._fields_ AttributeError: _fields_ is final) #211
• Pre-processor error on Windows system headers (VARARGS not implemented) #212

PRs/Patches

• Port ctypesgen to argparse #161
• Fix function macros with empty params list #189
• Fix conflicting names resolver being never called #193
• [Backport] Fix presumed no vs never typo #191
• [Backport] Prevent assignment of invalid struct fields (fixes #183) #205
• [Backport] Fix linking of relative modules and external preamble #206
• [Backport] Fix syntax errors on defines using __attribute__ ((...)) #207
• Make string auto-conversion optional & use leaner strings classes #177
• Update variadic functions code #192
• Replace custom c_ptrdiff_t impl with built-in ctypes.c_ssize_t #210

Footnotes

1. Unfortunately the readme/issue may be somewhat outdated/incomplete due to ongoing development, where there isn't really an easy way to keep it all up-to-date without having to spend too much time. ↩

[mara004]

Also an idea: You could divide this repo in two branches: One with our changes, where any future active development shall go to, while still backporting key fixes to the "oldschool" codebase, at least for a migration period.

Basically just what I'm currently doing with a fork and backport PRs such as #193, #189, #191, but fused into a single repo.
We wouldn't need to ditch the legacy codebase altogether if that's too much of a compatibility concern.

[nilason]

@mara004 I'm terribly sorry for not have had the time to study your work on this, but I'm glad you don't give up!

I'm thinking of make a v1.x release branch after any minor eventual changes to master, to be able to more freely push more experimental code to master (better I think than creating an experimental branch). What is your opinion on this?

[mara004]

Thanks, no worries!

I'm thinking of make a v1.x release branch after any minor eventual changes to master, to be able to more freely push more experimental code to master (better I think than creating an experimental branch). What is your opinion on this?

A v1 branch plus experimental master sounds fine to me.
I don't really mind how, just thought it might be good to keep an ability to push to the previous codebase.

[mara004]

BTW, in case anyone wants to merge our changes at large, the Readme would need to be updated as it talks about the fork, yet it would be good to retain the new info (e.g. tips & tricks section).
Also you might want to restore some tool files which I removed as we don't use/maintain them.

[mara004]

Here's a list of smaller fixes we might want to backport to the v1 codebase:

• struct __slots__ fix to prevent assignment of invalid fields
• ability to link relative modules, don't silently ignore import failure, remove bad POINTER override
• remove variadic functions wrapper, we can simply assign .argtypes (presumably fixes variadics on macOS arm64)
• handle __attribute__ ((...)) in defines (fixes cdefs.h syntax errors)
• smarter pre-processor default

I'm not too convinced if we have to do this, though.
Perhaps it might be better to just focus efforts on the new branch and avoid touching v1 unless there's an actual, reasoned demand?
I don't want to end up rewriting too much of the new base...

[mara004]

Three backports done: #205, #206, #207