AIX: build fails for xlc/xlC since new PEG parser #85387
Comments
|
As the bots were both running - based on gcc - this was not noticed immediately. bpo-40334 implements PEP-617, the new PEG parser for CPython. Using bisect I located: commit c5fc156 (HEAD, refs/bisect/bad) +++ the make status (abbreviated) is: +++ The complete make.out (stdout) is: After /bin/sh: 7405692 Segmentation fault(coredump) Stop. Sun Jul 5 11:23:39 UTC 2020 The above includes aixtools@x064:[/data/prj/python/git/py39-3.10]git checkout 458004b This still crashes with the message: I'll add more debug info in a followup - starting at: Previous HEAD position was a25f3c4 bpo-40334: Fix builds outside the source directory and regenerate autoconf files (GH-19667) |
aixtools
added
3.9
|
Well, the first step -s just showing where the segmentation fault occurs (in pegen.c). I am not really 'wiser' in what I should be looking at. I'll try adding a fprintf(stderr, ....) to see if I can figure out a bit more. For the experts: (dbx) where (dbx) registers (dbx) listi strncmp |
|
OK - merely added some fprintf statements. When it is working as expected, the k->type values seem to be between 500 and 535 - when it fails the k->type value is frequently 9 digits (e.g., 537120904) - and it seems to never become -1 -- which would end the loop and "RETURN NAME". One working example... NAMEPTR: 20089d44, name_len:6 strlen(name):4472 name.16s: token: 20030d30, k_type:500, strlen(k->str):6 k->str.16s: token: 20030d38, k_type:505, strlen(k->str):6 k->str.16s: token: 20030d4, k_type:508, strlen(k->str):6 k->str.16s: token: 20030d48, k_type:513, strlen(k->str):6 k->str.16s: RETURN k->type: 513 from k-str.16:import And a failed (abbreviated) that ends with the segmentation error: NAMEPTR: 20089d60, name_len:8 strlen(name):4444 name.16s: token: 2000f340, k_type:537120904, strlen(k->str):0 k->str.16s: token: 2000f348, k_type:271488524, strlen(k->str):8 k->str.16s: token: 2000f350, k_type:805316552, strlen(k->str):0 k->str.16s: token: 2000f358, k_type:368, strlen(k->str):11 k->str.16s: token: 2000f360, k_type:537120928, strlen(k->str):8 k->str.16s: token: 2000f368, k_type:0, strlen(k->str):61 k->str.16s: token: 2000f370, k_type:271482672, strlen(k->str):0 k->str.16s: ... token: 2000f738, k_type:0, strlen(k->str):0 k->str.16s: token: 2000f740, k_type:0, strlen(k->str):0 k->str.16s: token: 2000f748, k_type:0, strlen(k->str):0 k->str.16s: token: 2000f750, k_type:0, strlen(k->str):0 k->str.16s: token: 2000f758, k_type:0, strlen(k->str):0 k->str.16s: token: 2000f760, k_type:0, strlen(k->str):0 k->str.16s: token: 2000f768, k_type:0, strlen(k->str):0 k->str.16s: token: 2000f770, k_type:271810528, strlen(k->str):0 k->str.16s: token: 2000f778, k_type:0, strlen(k->str):0 k->str.16s: token: 2000f780, k_type:271810512, strlen(k->str):4 k->str.16s: token: 2000f788, k_type:0, strlen(k->str):0 k->str.16s: /bin/sh: 8847524 Segmentation fault(coredump) Hope this helps. I really do not know 'WHERE' it went wrong. I can only begin by guessing. So, expert guidance is appreciated! |
|
Unfortunately, I am having a hard time parsing your error description because is not immediate to distinguish:
Could you kindly provide more scoped information about these points so we can look into it? |
|
Michael, can you try with this patch: diff --git a/Parser/pegen.c b/Parser/pegen.c
index 53e3d49138..7faeec26ad 100644
--- a/Parser/pegen.c
+++ b/Parser/pegen.c
@@ -528,7 +528,7 @@ _get_keyword_or_name_type(Parser *p, const char *name, int name_len)
if (name_len >= p->n_keyword_lists || p->keywords[name_len] == NULL) {
return NAME;
}
- for (KeywordToken *k = p->keywords[name_len]; k->type != -1; k++) {
+ for (KeywordToken *k = p->keywords[name_len]; k != NULL && k->type != -1; k++) {
if (strncmp(k->str, name, name_len) == 0) {
return k->type;
}For whatever reasons, it seems that me are calling into _get_keyword_or_name_type with a name_len of 0. |
|
Iirc, my debug shows that k is not NULL, as k++ is not going to suddenly become smaller. And my debug shows it grows by a constant. My gut says the pointer to the base of the tokens is wrong, because the key types are such different values. Or do you know that those are valid types? |
|
I tried compiling Python 3.10 with XLC on AIX and I get a crash in a different place that does not involve the parser: _PyEval_EvalFrameDefault(tstate = 0x200764e0, f = 0x300b5190, throwflag = 0), line 427 in "object.h" |
|
Ok, I have investigated more and it seems that XLC is miscompiling the keyword list. For instance, if you add: index 9d3ac575df..70d431e6be 100644
--- a/Parser/pegen/parse.c
+++ b/Parser/pegen/parse.c
@@ -1,6 +1,6 @@
-// @generated by pegen.py from ./Grammar/python.gram
#include "pegen.h"
+// @generated by pegen.py from ./Grammar/python.gram
#if defined(Py_DEBUG) && defined(Py_BUILD_CORE)
extern int Py_DebugFlag;
#define D(x) if (Py_DebugFlag) x;
@@ -24735,7 +24735,12 @@ _PyPegen_parse(Parser *p)
// Initialize keywords
p->keywords = reserved_keywords;
p->n_keyword_lists = n_keyword_lists;
-
+ for (int s=0; s<p->n_keyword_lists;s++) {
+ KeywordToken *kk = p->keywords[s];
+ if (kk) {
+ printf("--> %i %.*s\n", s,s, kk->str);
+ }
+ }
// Run parser
void *result = NULL;
if (p->start_rule == Py_file_input) {Before crashing, this will print the following: --> 2 if Notice that the entry for size 8 is missing. I don't understand why is missing because the data is static and there is an entry there: static KeywordToken *reserved_keywords[] = { |
|
Ok, this is definitively something going on with XLC. This patch solves the segfault: diff --git a/Parser/pegen/parse.c b/Parser/pegen/parse.c
index 9d3ac575df..e5511bf815 100644
--- a/Parser/pegen/parse.c
+++ b/Parser/pegen/parse.c
@@ -1,6 +1,6 @@
-// @generated by pegen.py from ./Grammar/python.gram
-#include "pegen.h"
+#include "pegen.h"
+// @generated by pegen.py from ./Grammar/python.gram
#if defined(Py_DEBUG) && defined(Py_BUILD_CORE)
extern int Py_DebugFlag;
#define D(x) if (Py_DebugFlag) x;
@@ -9,8 +9,8 @@ extern int Py_DebugFlag;
#endif
static const int n_keyword_lists = 15;
static KeywordToken *reserved_keywords[] = {
- NULL,
- NULL,
+ (KeywordToken[]) {{NULL, -1}},
+ (KeywordToken[]) {{NULL, -1}},
(KeywordToken[]) {
{"if", 510},
{"in", 518},
@@ -65,11 +65,11 @@ static KeywordToken *reserved_keywords[] = {
{"nonlocal", 509},
{NULL, -1},
},
- NULL,
- NULL,
- NULL,
- NULL,
- NULL,
+ (KeywordToken[]) {{NULL, -1}},
+ (KeywordToken[]) {{NULL, -1}},
+ (KeywordToken[]) {{NULL, -1}},
+ (KeywordToken[]) {{NULL, -1}},
+ (KeywordToken[]) {{NULL, -1}},
(KeywordToken[]) {
{"__peg_parser__", 531},
{NULL, -1},
@@ -24735,7 +24735,6 @@ _PyPegen_parse(Parser *p)
// Initialize keywords
p->keywords = reserved_keywords;
p->n_keyword_lists = n_keyword_lists;
-
// Run parser
void *result = NULL;
if (p->start_rule == Py_file_input) {Is a C99 violation to have NULL elements in this array? (Michael, could you also double check that this patch solves it for you?) |
|
This is enough to reproduce the problem: #include <stdio.h>
typedef struct {
char *str;
int type;
} KeywordToken;static KeywordToken *the_array[] = { int main() {
for (int s=0; s<7;s++) {
KeywordToken *tok = the_array[s];
printf("--> %i\n",s);
if (tok == NULL) {
continue;
}
printf("--> %i %.*s\n", s,s, tok->str);
}
return 0;
}[cpython (3.9)]$ xlc check_bad.c -o check_bad But if you change it to: #include <stdio.h>
typedef struct {
char *str;
int type;
} KeywordToken;static KeywordToken *the_array[] = { int main() {
for (int s=0; s<7;s++) {
KeywordToken *tok = the_array[s];
if (tok == NULL) {
continue;
}
printf("--> %i %.*s\n", s,s, tok->str);
}
return 0;
}[cpython (3.9)]$ xlc check.c -o check |
|
Glad you figured it out! I doubt I would have. Thx!! On 06/07/2020 14:37, Pablo Galindo Salgado wrote:
|
Well, this is not over ;) We should confirm that this is either a bug in XLC or a violation of C99 |
|
On 06/07/2020 16:35, Pablo Galindo Salgado wrote:
probably a bug (that they might call a difference of implementation). I have been testing with xlc-v11; shall also try (your simplified
|
|
I don't believe that this is an XLC bug, but I suspect that it is undefined behavior / implementation-defined behavior. I suspect that this is tripping over AIX/XLC null behavior. AIX specifically and intentionally maps the first page of memory at address 0 to allow the compiler to speculate through NULL pointers. The compiler probably is speculating in this case and the second element is not defined. There is some option to disable this speculation in XLC. |
|
I tried check.c and check_bad.c using xlc-v11 (on my POWER6) - and the On the gcc119 host - using the v13 compiler, check_bad does not crash. I have seen lots of options today - wheile researching, so probably, Atm - testing "master" build using xlc-v11 and xlc-v13. On 06/07/2020 18:13, David Edelsohn wrote:
|
I was looking at the C99 standard but could not find anything that makes this undefined. Do you know what this construct is contradicting the standard on? |
|
Note: - two different systems, different HW, different OS levels. xlc-v11, master : commit deb0162 (HEAD Segmentation fault in _PyEval_EvalFrameDefault at line 941 in file +++++++++++++ xlc-v13 +++++++++++++ Looks about the same: master : commit aixtools@gcc119:[/home/aixtools/python/cpython]dbx ./python core Segmentation fault in _PyEval_EvalFrameDefault at line 1125 in file On 06/07/2020 18:57, Pablo Galindo Salgado wrote:
|
|
Maybe XLC was being overly aggressive with speculation and it now is fixed. I can't tell if Michael's earlier comment meant that it no longer crashes with XLC v16. |
|
Well, in any case, I will submit a patch today to the parser generator to substitute the NULL for single-element arrays which seems to work. Michael, could you confirm that this patch works for you: diff --git a/Parser/pegen/parse.c b/Parser/pegen/parse.c
index 9d3ac575df..e5511bf815 100644
--- a/Parser/pegen/parse.c
+++ b/Parser/pegen/parse.c
@@ -1,6 +1,6 @@
-// @generated by pegen.py from ./Grammar/python.gram
-#include "pegen.h"
+#include "pegen.h"
+// @generated by pegen.py from ./Grammar/python.gram
#if defined(Py_DEBUG) && defined(Py_BUILD_CORE)
extern int Py_DebugFlag;
#define D(x) if (Py_DebugFlag) x;
@@ -9,8 +9,8 @@ extern int Py_DebugFlag;
#endif
static const int n_keyword_lists = 15;
static KeywordToken *reserved_keywords[] = {
- NULL,
- NULL,
+ (KeywordToken[]) {{NULL, -1}},
+ (KeywordToken[]) {{NULL, -1}},
(KeywordToken[]) {
{"if", 510},
{"in", 518},
@@ -65,11 +65,11 @@ static KeywordToken *reserved_keywords[] = {
{"nonlocal", 509},
{NULL, -1},
},
- NULL,
- NULL,
- NULL,
- NULL,
- NULL,
+ (KeywordToken[]) {{NULL, -1}},
+ (KeywordToken[]) {{NULL, -1}},
+ (KeywordToken[]) {{NULL, -1}},
+ (KeywordToken[]) {{NULL, -1}},
+ (KeywordToken[]) {{NULL, -1}},
(KeywordToken[]) {
{"__peg_parser__", 531},
{NULL, -1},
@@ -24735,7 +24735,6 @@ _PyPegen_parse(Parser *p)
// Initialize keywords
p->keywords = reserved_keywords;
p->n_keyword_lists = n_keyword_lists;
-
// Run parser
void *result = NULL;
if (p->start_rule == Py_file_input) { |
|
Pablo, I can do that as well, if you don't have the time. |
Thanks! I had the PR ready, just needed some time to push it :) |
|
I also tested xlC v16, and then it just hung, for hours - while all of you were being more productive. I’ll kickoff my bot again, and see how it goes. Sent from my iPhone
|
|
Michael, could you check the latest master and 3.9 HEAD? If you don;t see the problem anymore, we can close this issue :) |
|
Pablo, on second thought, should we maybe change the assertion to |
Yup, although not sure how that can happen (but I agree that's the point of assertions *wink*) Can you prepare a PR? |
Done. |
|
I saw the mails last night and restarted my bot - it still fails. Yes - expecting 3.8 and 3.7 to build, but want to be sure. On 06/07/2020 23:34, Pablo Galindo Salgado wrote:
|
|
On 07/07/2020 11:12, Michael Felt wrote:
|
|
Master has to be something else then.... |
|
Here is the stack trace - still during initialization: And in "ceval.c, (dbx) list Segmentation fault in _PyEval_EvalFrameDefault at line 941 in file On 07/07/2020 12:45, Pablo Galindo Salgado wrote:
|
|
Yeah, this looks like something else. I am closing this issue then |
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
bugs.python.org fields:
The text was updated successfully, but these errors were encountered: