gh-119105: difflib: improve recursion for degenerate cases #119131

pulkin · 2024-05-17T21:11:50Z

Used this as a benchmark:

a = ["0123456789\n"] * 1_000
b = ["01234a56789\n"] * 1_000

from difflib import Differ
import timeit

print(timeit.timeit("list(Differ().compare(a, b))", number=1, globals=globals()))
# this exceeds recursion depth on my stock python distribution.
# once `difflib` is patched, it runs in approximately `1s`

Issue: difflib.py Differ.compare is too slow [for degenerate cases] #119105

cpython-cla-bot · 2024-05-17T21:11:53Z

All commit authors signed the Contributor License Agreement.

tim-one · 2024-05-18T23:48:15Z

You may have noticed that it's bugging you for a NEWS entry. Easiest is to click "Details" on the failing "bedevere/news" check, That will bring up a web page that asks you for the GitHub issue number, PR number, which part of the distribution is affected ("library", here), and type some text explaining the change at a high (user-oriented) level. When that's done, submit the page, and it will magically generate a new file with a crazy name, and commit it to this branch in the Misc/NEWS.d directory.

Lib/difflib.py

Co-authored-by: Tim Peters <tim.peters@gmail.com>

Misc/NEWS.d/next/Library/2024-05-19-12-25-36.gh-issue-119105.VcR4ig.rst

Co-authored-by: Tim Peters <tim.peters@gmail.com>

tim-one

Looks good to go! It's been a pleasure working with you - I hope you stick around 😄.

tim-one · 2024-05-19T18:14:28Z

BTW, I'm not sure whether it will let you "squash and merge" the change. If not, let me know and I'll do the commit.

pulkin · 2024-05-19T21:30:40Z

Just letting you know that I am fine with the merge. I am not allowed to merge.

Thanks for the guidance and discussion btw.

tim-one · 2024-05-19T21:49:23Z

All done! You're an official Python Committer™ now. Use your power only for good 😉

bedevere-bot · 2024-05-19T22:41:08Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM64 MacOS M1 Refleaks NoGIL 3.x has failed when building commit 0abf997.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/1368/builds/1028) and take a look at the build logs.
Check if the failure is related to this commit (0abf997) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/1368/builds/1028

Failed tests:

test_sqlite3

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 9, done.        
remote: Counting objects:  11% (1/9)        
remote: Counting objects:  22% (2/9)        
remote: Counting objects:  33% (3/9)        
remote: Counting objects:  44% (4/9)        
remote: Counting objects:  55% (5/9)        
remote: Counting objects:  66% (6/9)        
remote: Counting objects:  77% (7/9)        
remote: Counting objects:  88% (8/9)        
remote: Counting objects: 100% (9/9)        
remote: Counting objects: 100% (9/9), done.        
remote: Compressing objects:  12% (1/8)        
remote: Compressing objects:  25% (2/8)        
remote: Compressing objects:  37% (3/8)        
remote: Compressing objects:  50% (4/8)        
remote: Compressing objects:  62% (5/8)        
remote: Compressing objects:  75% (6/8)        
remote: Compressing objects:  87% (7/8)        
remote: Compressing objects: 100% (8/8)        
remote: Compressing objects: 100% (8/8), done.        
remote: Total 9 (delta 1), reused 2 (delta 1), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to '0abf997e75bd3a8b76d920d33cc64d5e6c2d380f'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 0abf997e75 gh-119105: difflib: improve recursion for degenerate cases (#119131)
Switched to and reset branch 'main'

In file included from ./Modules/tkappinit.c:17:
In file included from /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/tk.h:99:
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:131:21: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*free_private)();  /* called to free private storage */
                           ^
                            void
In file included from ./Modules/_tkinter.c:52:
In file included from /opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/tk.h:99:
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:131:21: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*free_private)();  /* called to free private storage */
                           ^
                            void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:334:33: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        struct _XImage *(*create_image)();
                                       ^
                                        void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:334:33: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        struct _XImage *(*create_image)();
                                       ^
                                        void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:453:23: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        XID (*resource_alloc)(); /* allocator function */
                             ^
                              void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:471:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*synchandler)();   /* Synchronization handler */
                          ^
                           void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:453:23: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        XID (*resource_alloc)(); /* allocator function */
                             ^
                              void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:496:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Bool (*event_vec[128])();  /* vector for wire to event */
                              ^
                               void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:471:20/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:: 497warning: :a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*synchandler)();   /* Synchronization handler */
25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Status (*wire_vec[128])(); /* vector for event to wire */
                               ^
                                void
                          ^
                           void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:509:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Bool (**error_vec)();      /* vector for wire to error */
                          ^
                           void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:522:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*savedsynchandler)(); /* user synchandler when Xlib usurps */
                               ^
                                void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:496:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Bool (*event_vec[128])();  /* vector for wire to event */
                              ^
                               void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:497:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Status (*wire_vec[128])(); /* vector for event to wire */
                               ^
                                void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:1053:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
typedef void (*XIMProc)();
                       ^
                        void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:509:20: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        Bool (**error_vec)();      /* vector for wire to error */
                          ^
                           void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:522:25: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
        int (*savedsynchandler)(); /* user synchandler when Xlib usurps */
                               ^
                                void
/opt/homebrew/Cellar/tcl-tk/8.6.13_5/include/tcl-tk/X11/Xlib.h:1053:24: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
typedef void (*XIMProc)();
                       ^
                        void
9 warnings generated.
9 warnings generated.

make: *** [buildbottest] Error 2

bedevere-bot · 2024-05-19T23:14:15Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot s390x Fedora LTO + PGO 3.x has failed when building commit 0abf997.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/545/builds/5934) and take a look at the build logs.
Check if the failure is related to this commit (0abf997) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/545/builds/5934

Failed tests:

test_free_threading

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 9, done.        
remote: Counting objects:  11% (1/9)        
remote: Counting objects:  22% (2/9)        
remote: Counting objects:  33% (3/9)        
remote: Counting objects:  44% (4/9)        
remote: Counting objects:  55% (5/9)        
remote: Counting objects:  66% (6/9)        
remote: Counting objects:  77% (7/9)        
remote: Counting objects:  88% (8/9)        
remote: Counting objects: 100% (9/9)        
remote: Counting objects: 100% (9/9), done.        
remote: Compressing objects:  12% (1/8)        
remote: Compressing objects:  25% (2/8)        
remote: Compressing objects:  37% (3/8)        
remote: Compressing objects:  50% (4/8)        
remote: Compressing objects:  62% (5/8)        
remote: Compressing objects:  75% (6/8)        
remote: Compressing objects:  87% (7/8)        
remote: Compressing objects: 100% (8/8)        
remote: Compressing objects: 100% (8/8), done.        
remote: Total 9 (delta 1), reused 2 (delta 1), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to '0abf997e75bd3a8b76d920d33cc64d5e6c2d380f'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at 0abf997e75 gh-119105: difflib: improve recursion for degenerate cases (#119131)
Switched to and reset branch 'main'

configure: WARNING: no system libmpdecimal found; unable to build _decimal

find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
find: ‘build’: No such file or directory
make[2]: [Makefile:3113: clean-retain-profile] Error 1 (ignored)
/home/dje/cpython-buildarea/3.x.edelsohn-fedora-z.lto-pgo/build/Lib/test/test_decimal.py:5919: UserWarning: C tests skipped: no module named _decimal.
  warnings.warn('C tests skipped: no module named _decimal.',
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘doParseXmlDecl’ at ./Modules/expat/xmltok.c:1192:13:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./Modules/expat/xmltok.c: In function ‘doParseXmlDecl’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1070:9:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1076:12:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1076:12:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1083:9:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1083:9:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1110:7:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1123:9:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1123:9:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^~~
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘doParseXmlDecl’ at ./Modules/expat/xmltok.c:1192:13:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^
./Modules/expat/xmltok.c: In function ‘doParseXmlDecl’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1070:9:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1076:12:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1076:12:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1083:9:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1083:9:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1110:7:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1123:9:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^
In function ‘utf8_toUtf8’,
    inlined from ‘toAscii’ at ./Modules/expat/xmltok.c:1038:3,
    inlined from ‘parsePseudoAttribute’ at ./Modules/expat/xmltok.c:1123:9:
./Modules/expat/xmltok.c:392:5: warning: ‘memcpy’ writing 2 or more bytes into a region of size 1 overflows the destination [-Wstringop-overflow=]
  392 |     memcpy(*toP, *fromP, bytesToCopy);
      |     ^
./Modules/expat/xmltok.c: In function ‘parsePseudoAttribute’:
./Modules/expat/xmltok.c:1036:8: note: destination object ‘buf’ of size 1
 1036 |   char buf[1];
      |        ^

make: *** [Makefile:2240: buildbottest] Error 2

tim-one · 2024-05-19T23:35:06Z

I believe the buildbot failures are spurious "false positives" for this PR. One is failing a threading test, and the other sqlite3. difflib has no plausible connection to either. When a PR is "really" at fault, some dozens of these buildbot failures are generated.

…hon#119131) Code from https://github.com/pulkin, in PR python#119131 Greatly speeds `Differ` when there are many identically scoring pairs, by splitting the recursion near the inputs' midpoints instead of degenerating (as now) into just peeling off the first two lines. Co-authored-by: Tim Peters <tim.peters@gmail.com>

pulkin added 2 commits May 17, 2024 22:59

difflib.py: fix recursion branching for degenerate cases

9bb454c

difflib.py: grammar

a159954

bedevere-app bot added the awaiting review label May 17, 2024

bedevere-app bot mentioned this pull request May 17, 2024

difflib.py Differ.compare is too slow [for degenerate cases] #119105

Closed

pulkin added 2 commits May 18, 2024 10:34

difflib.py: weights-free implementation

81b7dae

difflib.py: rm left-over

2184911

tim-one self-requested a review May 18, 2024 23:39

tim-one reviewed May 18, 2024

View reviewed changes

Lib/difflib.py Show resolved Hide resolved

pulkin and others added 5 commits May 19, 2024 14:18

difflib.py: even simpler impl

f9e3480

📜🤖 Added by blurb_it.

b0be97f

difflib.py: add a desc

bea603e

Co-authored-by: Tim Peters <tim.peters@gmail.com>

📜🤖 Added by blurb_it.

df83f50

news: attempt fix build

0953ede

tim-one reviewed May 19, 2024

View reviewed changes

Misc/NEWS.d/next/Library/2024-05-19-12-25-36.gh-issue-119105.VcR4ig.rst Outdated Show resolved Hide resolved

difflib.py: NEWS message

7237b48

Co-authored-by: Tim Peters <tim.peters@gmail.com>

tim-one approved these changes May 19, 2024

View reviewed changes

bedevere-app bot added awaiting merge and removed awaiting review labels May 19, 2024

python deleted a comment from bedevere-app bot May 19, 2024

tim-one linked an issue May 19, 2024 that may be closed by this pull request

difflib.py Differ.compare is too slow [for degenerate cases] #119105

Closed

tim-one self-assigned this May 19, 2024

tim-one merged commit 0abf997 into python:main May 19, 2024
35 checks passed

bedevere-app bot removed the awaiting merge label May 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-119105: difflib: improve recursion for degenerate cases #119131

gh-119105: difflib: improve recursion for degenerate cases #119131

pulkin commented May 17, 2024 •

edited by bedevere-app bot

Loading

cpython-cla-bot bot commented May 17, 2024 •

edited

Loading

tim-one commented May 18, 2024

tim-one left a comment

tim-one commented May 19, 2024 •

edited

Loading

pulkin commented May 19, 2024

tim-one commented May 19, 2024

bedevere-bot commented May 19, 2024

bedevere-bot commented May 19, 2024

tim-one commented May 19, 2024

gh-119105: difflib: improve recursion for degenerate cases #119131

gh-119105: difflib: improve recursion for degenerate cases #119131

Conversation

pulkin commented May 17, 2024 • edited by bedevere-app bot Loading

cpython-cla-bot bot commented May 17, 2024 • edited Loading

tim-one commented May 18, 2024

tim-one left a comment

Choose a reason for hiding this comment

tim-one commented May 19, 2024 • edited Loading

pulkin commented May 19, 2024

tim-one commented May 19, 2024

bedevere-bot commented May 19, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

bedevere-bot commented May 19, 2024

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

tim-one commented May 19, 2024

pulkin commented May 17, 2024 •

edited by bedevere-app bot

Loading

cpython-cla-bot bot commented May 17, 2024 •

edited

Loading

tim-one commented May 19, 2024 •

edited

Loading