Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

global: Remove the STATIC macro. #13763

Merged

Conversation

projectgus
Copy link
Contributor

@projectgus projectgus commented Feb 27, 2024

@dpgeorge has said for a while that he wants to get rid of the STATIC macro. No time like the present!

The macro is fully removed rather than deprecated. I think this is reasonable as it's a relatively quick refactor to clean up from, but I have a script (see below) to post a heads-up to everyone with an open PR that adds the STATIC macro.

If that's not acceptable then I think we could add a CI check that blocks addition of STATIC, but keep the macro defined otherwise. But I think this way is probably less fiddly in some ways (doesn't leave contributors bouncing off the CI.)

Methodology

  1. git ls-files | egrep '\.[ch]$' | xargs sed -Ei "s/(^| )STATIC($| )/\1static\2/"
  2. Do some manual cleanup in the diff by searching for the word STATIC in comments and changing those back.
  3. git-grep STATIC docs/, manually fixed those cases
  4. rg -t python STATIC, manually fixed the codegen lines that used STATIC.
  5. A few files in renesas-ra port flagged codespell so there are some spelling fixes on top of everything else.
  6. Few one-off cases (coverage.cpp file, a lone STATIC in a parenthesis, etc.) were caught by CI.

Automated heads-up

I have knocked up a quick script to leave a comment on every PR that seems to add STATIC in the diff, letting them know that when they rebase they'll need to change it to static.

(Of course, there will also be some inevitable merge conflicts from changed lines. There isn't really a way around that, apart from not moving away from the macro.)

Follow-up work

Separate PR georgerobotics/cyw43-driver#109 to remove STATIC from cyw43-driver repo. Once merged then can revert the commit in this branch that manually sets the macro when building those files.

This work was funded through GitHub Sponsors.

@projectgus
Copy link
Contributor Author

projectgus commented Feb 27, 2024

@jimmo The comment script took me just under 30 minutes. Not sure if that counts as good automation or not? 😁

(Leaving hundreds of comments would take hours but I would never do that, the real time saved is PR authors knowing in advance that when they rebase they'll have to apply the change.)

Copy link

github-actions bot commented Feb 27, 2024

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS

Copy link

codecov bot commented Feb 27, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.39%. Comparing base (90e5178) to head (decf8e6).

Additional details and impacted files
@@            Coverage Diff             @@
##           master   #13763      +/-   ##
==========================================
+ Coverage   98.36%   98.39%   +0.02%     
==========================================
  Files         161      161              
  Lines       21084    21078       -6     
==========================================
  Hits        20739    20739              
+ Misses        345      339       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jimmo
Copy link
Member

jimmo commented Feb 27, 2024

@jimmo The comment script took me just under 30 minutes. Not sure if that counts as good automation or not? 😁

@projectgus Excellent automation, would automate again.

Thanks for doing this PR :)

@projectgus projectgus marked this pull request as ready for review February 27, 2024 06:06
@jimmo jimmo mentioned this pull request Feb 27, 2024
6 tasks
@dpgeorge
Copy link
Member

The original reason for this STATIC macro was to define it to nothing so that all static functions become global functions, and therefore visible to certain debug tools. From commit d5df6cd:

Some tools do not support local/static symbols (one example is GNU ld map file).
Exposing all functions will allow to do detailed size comparisons, etc.

Personally I have never used this feature. And with the use of LTO and heavy inline optimisation, analysing size of individual functions when they are not static is not a good representation of the size of code when fully optimised.

So I don't really see any use for this STATIC macro. And it's simpler not to have the macro, ie just use static as-is, because then you know exactly what it's doing. Eg for newcomers they don't have to learn what the STATIC macro is.

One other point in favour of removing it, is that it stops bugs with STATIC inline, which should always be static inline.

@projectgus
Copy link
Contributor Author

@tannewt @dhalbert Heads-up about something with potential to cause some transient pain in the next merge. :)

@robert-hh
Copy link
Contributor

It will likewise be pain for everyone with open PRs, I guess.

@robert-hh
Copy link
Contributor

@projectgus For the automation script I used a slightly modified version and replaced
git ls-files with git show --pretty="" --name-only <first_commit>..<last_commit> | sort | uniq, with <first_commit> and <last_commit> being the commit id's of the PR. That way, only the files are touched that belong to the PR. Otherwise all files of the repository will be processed.

Gadgetoid added a commit to pimoroni/pimoroni-pico that referenced this pull request Feb 29, 2024
Gadgetoid added a commit to Gadgetoid/micropython-ulab that referenced this pull request Feb 29, 2024
@Gadgetoid
Copy link
Contributor

Thank you for posting your methodology, it helped me prepare my codebase and raise a PR against ulab - v923z/micropython-ulab#664

And a patch against pimoroni-pico - pimoroni/pimoroni-pico#906

AIUI all downstream code can pretty much make this change immediately. In our case (pimoroni-pico) we pretty much universally expect STATIC to resolve to static and, afaik, have never used the mentioned technique for debugging since most of our Python bindings are just glue to underlying classes anyway.

It will likewise be pain for everyone with open PRs, I guess.

Pain is good. Separates the serious contributors from the drive bys 😆

@robert-hh
Copy link
Contributor

Pain is good.

Pain is pain. Besides that, I expect quite a few merge conflicts, independent from how the change will be made. - If a branch is not processed according to this PR before attempting a rebase, there will be merge conflicts at the new commits when rebasing. Git cannot tell whether STATIC or static is right, and will eventually even include adjacent lines into the conflict's span.

  • If a branch is processed according to this PR before rebase, it could only work if there are no changes to the master branch after applying the global replace. If there have been such, there will by conflicts is the base part of the code.

There are some ways to avoid that, like:

  • deliberately change STATIC to static in the files changed in a branch before rebasing to the changed master. That must be done in reverse order from the newest to the oldest commit.
  • create a new branch for a feature , do a bulk change of the old branch for this feature and cherry-pick the commits from the old branch to the new one.

Obviously, code that did not use the STATIC macro is not affected.

@Gadgetoid
Copy link
Contributor

Gadgetoid commented Feb 29, 2024

According to a grep of every open PR, those affected include (129/342):

Edit: I have reformatted this list into a table and drawn attention to PRs from stakeholders in this thread.

User PR Title
🧙 dpgeorge #13764 stm32: Add support for analog-only "_C" pins.
tonyzhangnxp #13755 ports/mimxrt:Add thread support.
miketeachman #13727 esp32/machine_i2s: Integrate new I2S IDF driver.
felixdoerre #13689 extmod/network: Implement IPv6 API to set and retrieve nic configuration.
Walkline80 #13658 esp32/modsmartconfig: Add smartconfig module.
🧙 robert-hh #13630 stm32/ethernet: A few improvements.
🧙 dpgeorge #13583 webassembly: Add proxying between Python and JavaScript objects, and change top-level interface for PyScript use
🧙 projectgus #13572 py/stream: Check for stream read function returning too many bytes.
Gadgetoid #13470 ports/rp2: Add a means to set mass-storage filesystem label
nxp-yilin #13429 ports: mcx: Initial port for MCXN947 series MCU.
klondi #13390 Add a BufferedReader and allow BufferedWriter to handle partial writes and errors after some data was written.
nickovs #13378 ports/rp2: Provide direct memory access to UART FIFOs.
cwalther #13340 nrf: Implement time.time() and machine.RTC
PepperoniPingu #13281 machine_i2c: Add I2C bus clear support.
ricksorensen #13276 esp32/modesp32.c: Add mcu_temperature() function for C3/S2/S3 devices.
andrewleech #13011 examples/usercmodule: Add finaliser to cexample.
imnotjames #13000 extmod/asyncio: Add Task methods from CPython
BottleRocketeer #12949 extmod/modbtree: Add checks for already-closed database.
stinos #12918 py/objtype: Validate super() arguments.
agatti #12853 ports: Add RISC-V 32 bit QEMU port.
stinos #12810 windows: Implement socket/ssl modules
DvdGiessen #12780 extmod/modssl_mbedtls: Implement SSLSession support.
DvdGiessen #12732 esp32/modesp32: Implement idf_task_stats().
andrewleech #12722 stm32 / spiflash: Support defining exact flash chip(s) connected.
karlp #12650 ports/esp32: pwm: prevent duty glitch on init
🧙 jimmo #12644 py/mpconfig.h: Finer-grained super-opt setting.
mattytrentini #12540 ports/stm32/boards: Add WEACT_STM32H743 board.
🧙 projectgus #12487 py/modmicropython: Add micropython.memmove() and micropython.memset().
🧙 robert-hh #12464 samd: Support WiFi using external esp32 based modules.
xuancong84 #12429 Super enhancement for ESP8266: software-initiated MicroPython base firmware update from image file on LFS2 file-system and dynamic frozen modules
🧙 robert-hh #12347 mimxrt: Add Quadrature Encoder and Pulse Counter classes.
🧙 jimmo #12346 esp32: Add machine.Counter and machine.Encoder.
IhorNehrutsa #12345 esp32/MCPWM: Add motor control MCPWM driver.
IhorNehrutsa #12331 port/esp32: Add CAN(TWAI) driver.
kwagyeman #12324 ports/mimxrt: Add machine.CAN driver.
IhorNehrutsa #12293 esp32/machine_pin: Add mode and pull in machine_pin_print() as repr() function.
IhorNehrutsa #12291 extmod/machine_signal: Add signal_print() as repr() function.
IhorNehrutsa #12155 esp32/machine_timer: Add find free timer id.
mbedNoobNinja #12047 ports/renesas-ra: Ehternet support for VK-RA6M5
arbrauns #12008 py/persistentcode: Store constants more efficiently
fvstory #11880 ports/esp32:Add modesp32s3(special lcd_cam for esp32s3)
🧙 jimmo #11723 mpy-cross: WASM target and Python wheel.
dmazzella #11679 py/objint.c: Add support for int.bit_length().
ThinkTransit #11672 esp32/modesp32: Add temperature method for S2,C3 chips.
ThinkTransit #11572 ports/esp32/ulp: Initial draft PR for RISCV ULP support S2/S3.
DavidEGrayson #11500 py/stream.c: Implemented truncate().
cwalther #11430 Enable more features in the embed port
jaenrig-ifx #11365 New port for Infineon PSoC6 microcontrollers
jadonk #11360 Minor Zephyr port features
DavidEGrayson #11244 py/modsys.c: Add sys._exc_traceback.
keredson #11183 add set_stream to DecompIO
MaureenHelm #10859 zephyr: Construct Pin object with a port instance number.
IhorNehrutsa #10854 esp32/PWM: Reduce inconsitencies between ports.
IhorNehrutsa #10852 ESP32: Add DAC deinit() method.
dimkr #10783 extmod/modussl_mbedtls: Add closenotify() method
mbedNoobNinja #10752 ports/renesas-ra/boards/VK-RA6M3: Another New Board.
LordFlashmeow #10724 py/objdeque: Expand implementation to be doubly-ended
🧙 robert-hh #10607 ports/RTC: Attempt to reduce the inconsistencies between the port's RTC implementation.
mbedNoobNinja #10595 ports/renesas-ra/boards/VK-RA4W1: New board.
RetiredWizard #10400 ports/nrf: Add RTC support to NRF ports.
damz #10062 extmod/modussl_mbedtls: Wire in support for DTLS
laurensvalk #9997 py/runtime: Avoid crash on calling members of uninitialized native type.
Andrei-Pozolotin #9685 ports/esp32/esp32_rmt.c: #7015 ISR-driven pulse generator in RMT
glenn20 #9644 moduos: Add os.utime() support to vfs layer and LFS2, FAT and posix filesystems.
🧙 robert-hh #9624 samd/adc_dac: Implememt adc.read_timed() and dac.write_timed().
🧙 projectgus #9497 tinyusb: Support USB device drivers written in Python
andrewleech #9458 py/ringbuf: Add micropython.ringbuffer() interface for general use.
H-Grobben #9367 SMT32G4: Add USB, QPSI and AEMICS Board PYglet
Molorius #9356 shared/tinyusb: (just rp2 for now) change USB device settings and add HID
andrewleech #9309 stm32/btstack: Add support for btstack on WB55 / rfcore.
andrewleech #9046 stm32/wb55: Add usb/bluetooth transparent mode to stm32wb55 via native module.
chihosin #8995 ports/esp32: Fix ESP32-C3 deep/light sleep wake on GPIOs support.
🧙 jimmo #8945 extmod/modbluetooth: Remove non-sync event support.
🧙 jimmo #8922 py/builtinimport.c: Implement a "frozen overlay" using the filesystem.
andrewleech #8767 Improve sys.settrace to help support bdb/pdb debugging
IhorNehrutsa #8766 ESP32: Add Quadrature Encoder and Pulse Counter classes.
🧙 projectgus #8744 mpremote: Support bytecode raw paste for 'mpremote run module.mpy'
r4d10n #8637 ports/wch: Add support for WCH CH32V307.
Leah1115 #8593 py/modmath: New function math hypot.
RayaneCTX #8503 py: Implement decorator syntax relaxation from CPython 3.9.
🧙 dpgeorge #8381 Add VfsMap filesystem, mpremote deploy-mapfs, and ability to import .mpy files from ROM (WIP)
jbbjarnason #8331 Math.gcd new function
andrewleech #8318 nrf/bluetooth: Add support for nimble based ubluetooth.
🧙 dpgeorge #8270 stm32/machine_i2s: Add support for I2S on H7 MCUs (WIP)
harbaum #8253 Proposal: Esp32 machine.PCNT through Encoder.py and Counter.py
Romaric-RILLET #8236 mimxrt: Allow to choose the pin to use for miso and cs
alphaFred #8229 mimxrt/mboot: Adds bootloader support.
andrewleech #8027 extmod/modbluetooth: Add gap_indicate_service_changed.
karfas #7990 esp32/modmachine: wake_ext1_pins() returns which EXT1 pins caused wakeup.
andrewleech #7845 extmod/modbluetooth: Add gap_unpair command.
theidealist #7828 extmod/modframebuf.c: Expose width(), height(), format(), and stride() accessors on framebuf.Framebuffer
andrewleech #7796 windows/thread: Add support for win32 micropython _thread.
MrJake222 #7784 ports/esp8266: Software Serial (SoftUART) support.
andrewleech #7781 windows/bluetooth: Add support for nimble BLE to windows port.
caco3 #7725 Add support to read/write a single block in the RTC Memory
andrewleech #7703 stm32/wb55: Add usb/bluetooth transparent mode to stm32wb55.
eydam-prototyping #7562 ports/esp32/modmachine: Callbacks for system events.
marcidy #7526 Introduce Non-blocking wifi scan for ESP32
nickovs #7512 AES implementation without a TLS or SSL library
karfas #7133 esp32/machine_rtc: ESP32 machine.RTC().usermem() returning a bytearray.
ekondayan #7048 esp32/ota: Implement ESP-IDF OTA functionality.
steven-michalske #6963 stm32/can: Add support for a software based CAN message buffer.
EddieParis #6946 Add option in DHT driver to do the measure without blocking interrupts
🧙 dpgeorge #6668 Add support for async generators (PEP525)
zsquareplusc #6665 py/builtinimport: support relative import in custom import callbacks
IhorNehrutsa #6639 ESP32: New hardware PCNT Counter() and Encoder().
irsla #6482 modify deepsleep to allow wakeup on X1 or X18 on a STM32F7 (SF2/3/6W)
fgervais #6408 esp32: Expose touch_pad_get/set_meas_time() in python
nickovs #6389 Support for AES GCM mode
jonathanhogg #6263 extmod/modframebuf: FrameBuffer text scaling
🧙 dpgeorge #6125 extmod: implement basic "uevent" module (RFC, WIP)
andrewleech #6080 ports/unix: Add full uos.dupterm support.
seiuvwcdhol #6044 Fixed DHT timing error, Issue #5848
🧙 dpgeorge #6024 py/parsenum: Implement exact float parsing using integer mpz (WIP)
🧙 dpgeorge #6008 lib/utils/pyexec: improve pyexec so it can be used by unix (WIP)
tve #5973 esp32/time: make utime conform to CPython
🧙 jimmo #5926 stm32/profiler.c: Add gprof-compatible profiling for STM32.
tve #5711 RFC #2: Mechanism for allowing dynamic native modules to call port-specific functionality.
MrSurly #5542 Allow (unpadded) SPI transfers < 8bits
tschmid #5514 esp32/DAC: Add cosine generator capability
Jongy #5482 RFC: Linux kernel port
tve #5473 ESP32: support dynamic freq scaling and wifi power save
mcauser #5452 esp32: Add Sigma-Delta peripheral
esmil #5449 Add GD32VF103 port
🧙 jimmo #5195 py/gc: Print fragmentation stats in micropython.mem_info(1)
glennrub #5104 nrf/nfc: Add basic support for NFC tag 2.
🧙 dpgeorge #5025 [proof of concept] Implement parts of the core in Python bytecode
volodymyrlut #4464 [Fixed backport]: Add WPS and netstatus
livius2 #3583 Simple font size scaling for framebuf

This was achieved by fetching a .patch file for every open PR and running:

grep -lr "+STATIC" patches/ > affected.txt

Then wrangling the result into a list of URLs.

@robert-hh
Copy link
Contributor

That's quie a number, even if some of them can be considered as unmaintained or duplicates.

@Gadgetoid
Copy link
Contributor

That's quie a number, even if some of them can be considered as unmaintained or duplicates.

I concur! Earliest is from January 2018!! Is there a point at which PRs should be considered stale and closed for sanity's sake?

But either way, the potential impact is significant 😬

@robert-hh
Copy link
Contributor

Is there a point at which PRs should be considered stale and closed for sanity's sake

That would be a lot of work, looking into each of them. Damien closes sometimes some. And obviously, when they address an issue which is fixed otherwise. There could be a kind of timeout. Usually after a while there are changes to the main line which makes simple rebase impossible. If then the PR is not updated, it cannot be merged. So a check for the last change date could be a means.

@tannewt
Copy link
Sponsor

tannewt commented Feb 29, 2024

@tannewt @dhalbert Heads-up about something with potential to cause some transient pain in the next merge. :)

No concern for me. It is an easy change to do and a win for consistency.

Waiting to do it will just lead to more code that needs to be updated. Old PRs will have to be updated either way.

@robert-hh
Copy link
Contributor

Waiting to do it will just lead to more code that needs to be updated.

The discussion is not about waiting. It's just about the side effects and ways to proceed. Of course, who ever makes a PR has to maintain it.

v923z pushed a commit to v923z/micropython-ulab that referenced this pull request Feb 29, 2024
@projectgus projectgus force-pushed the refactor/remove_static_macro branch from a66b830 to a61bc35 Compare March 4, 2024 23:48
@projectgus
Copy link
Contributor Author

projectgus commented Mar 4, 2024

Rebased, checked no new STATIC snuck in.

@Gadgetoid Thanks for running through that and generating the PR table. I am planning to do something similar by automatically commenting on these PRs after merge (script linked in description), but it is good to provide the early link back notification as well! 🙏

Encouraging to see so much proactive removal already!

@robert-hh
Copy link
Contributor

Rebased, checked no new STATIC snuck in.

After making the global replace, did you have to manually revert some of them?

@projectgus
Copy link
Contributor Author

Have run the reminder script, as evidenced by the many link backs above. 😅

@dpgeorge
Copy link
Member

dpgeorge commented Mar 8, 2024

Have run the reminder script

Thanks! Let's see how this goes...

@robert-hh
Copy link
Contributor

At least that was quite a notification Tsunami.

Gadgetoid added a commit to pimoroni/pimoroni-pico that referenced this pull request Mar 8, 2024
Gadgetoid added a commit to pimoroni/pimoroni-pico that referenced this pull request Mar 8, 2024
ricksorensen added a commit to ricksorensen/micropython that referenced this pull request Mar 12, 2024
ricksorensen added a commit to ricksorensen/micropython that referenced this pull request Mar 13, 2024
Signed-off-by: Rick Sorensen <rick.sorensen@gmail.com>
yasnim24 added a commit to yasnim24/micropython that referenced this pull request Mar 17, 2024
See 'global: Remove the STATIC macro.':
micropython#13763

Signed-off-by: Yasmin Bosch <development@yasmin-bosch.de>
Gadgetoid added a commit to pimoroni/pimoroni-pico that referenced this pull request Apr 11, 2024
Gadgetoid added a commit to pimoroni/pimoroni-pico that referenced this pull request Apr 11, 2024
Gadgetoid added a commit to pimoroni/pimoroni-pico that referenced this pull request Apr 17, 2024
Gadgetoid added a commit to pimoroni/pimoroni-pico that referenced this pull request Jun 3, 2024
Gadgetoid added a commit to pimoroni/pimoroni-pico that referenced this pull request Jun 3, 2024
Gadgetoid added a commit to pimoroni/pimoroni-pico that referenced this pull request Jun 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants