Unexpected code in ReNumber #211
Comments
|
|
It must be an old bug. I got many f's with FORM 4.1 (Oct 25 2013). |
|
I’ll go after it. Specially with the crash (with 201 instead of 2001).
Jos
… On 30 jun. 2017, at 17:12, Takahiro Ueda ***@***.***> wrote:
It must be an old bug. I got many f's with FORM 4.1 (Oct 25 2013).
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub <#211 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFLxEvvcV9C8YRTsSYarHFUENpEZ9tVSks5sJRBagaJpZM4OKqxS>.
|
|
Great, that seems to work. Now, if I add to the first module: and to the second module: form has no problems, but tform -w4 crashes about 60% of the time, with various errors: EDIT: For this, at some point before the crash, valgrind gives |
|
Running gives me Now 100% it occurs (though it is not in a worker). |
|
Looks quite useful.
I spent a few hours yesterday on this stuff and the main conclusion was that there
was a counting error in the number of patches, because somewhere an end of expression
seems to not have been seen, unless a patch was written over another patch
instead of its intended spot.
But with expressions this size it is not easy to find what causes it.
In the example, it all played in a single worker. Same with the example
that you give. The dollar expands completely inside one worker.
Jos
… On 1 jul. 2017, at 21:39, Takahiro Ueda ***@***.***> wrote:
Running valgrind tvorm -w2 for
#:FilePatches 4
#:SubTermsInSmall 64
#:SubLargePatches 8
#:SubFilePatches 2
CF f;
#define NTERMS "5000"
#$x = <f(1)>+...+<f(`NTERMS')>;
.end
gives me
StageSort in thread 0
StageSort in thread 0
==11421== Syscall param read(buf) points to unaddressable byte(s)
==11421== at 0x32E1A0E82D: ??? (in /lib64/libpthread-2.12.so)
==11421== by 0x51C1B3: Uread (unixfile.c:134)
==11421== by 0x50B031: ReadFile (tools.c:1158)
==11421== by 0x42CF56: FillInputGZIP (compress.c:694)
==11421== by 0x4E95DB: PutIn (sort.c:1260)
==11421== by 0x4ED90D: MergePatches (sort.c:3656)
==11421== by 0x4EFB3C: EndSort (sort.c:1066)
==11421== by 0x4304D2: CatchDollar (dollar.c:117)
==11421== by 0x4B8301: PreProcessor (pre.c:1050)
==11421== by 0x4F4A43: main (startup.c:1605)
==11421== Address 0x37e9c640 is not stack'd, malloc'd or (recently) free'd
==11421==
0FillInputGZIP: Read error during uncompressed sort.
0++Reading 4295000 bytes at position 106280
PutIn: We have RetCode = FFFFFFFFFFFFFFFF while reading 418958 bytes
Program terminating in thread 0 at test.frm Line 7 -->
==11421== Invalid read of size 4
==11421== at 0x50F5F8: Crash (tools.c:3711)
==11421== by 0x4F4070: Terminate (startup.c:1707)
==11421== by 0x4E9618: PutIn (sort.c:1266)
==11421== by 0x4ED90D: MergePatches (sort.c:3656)
==11421== by 0x4EFB3C: EndSort (sort.c:1066)
==11421== by 0x4304D2: CatchDollar (dollar.c:117)
==11421== by 0x4B8301: PreProcessor (pre.c:1050)
==11421== by 0x4F4A43: main (startup.c:1605)
==11421== Address 0x0 is not stack'd, malloc'd or (recently) free'd
Now 100% it occurs (though it is not in a worker).
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub <#211 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFLxEtCiO7VcmLqXwdn5Sm8HNEc_NLuAks5sJqBXgaJpZM4OKqxS>.
|
|
The minimum seems to be 129 terms, but unfortunately it doesn't crash in spite of the memory bug. |
|
Two things:
In the following log, is The corresponding |
|
The first example causes memory leaks: https://travis-ci.org/tueda/form/jobs/249641044#L4542. (Should we reopen the issue?) |
|
I guess yes.
I am still looking at the “GZIP…..” and sometimes it crashes reading, sometimes the reading
goes ‘fine’ but it crashes in ReNumber. I have also put a print somewhere else and sometimes
it crashes there. But it seems all related.
The bug is extremely crazy though.
It crashes in the example in the first module, but only if the second module is present and
has code in it. ?????????
Jos
… On 3 jul. 2017, at 16:42, Takahiro Ueda ***@***.***> wrote:
The first example causes memory leaks: https://travis-ci.org/tueda/form/jobs/249641044#L4542 <https://travis-ci.org/tueda/form/jobs/249641044#L4542>. (Should we reopen the issue?)
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub <#211 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFLxEn-jvxiiLDtMTeNkh0GK-inVs1eoks5sKP3PgaJpZM4OKqxS>.
|
|
Here is another spam of valgrind log (vorm) with increasing It is very strange that |
|
What this seems to indicate that somehow the function/dollar POsize is, under certain
circumstances, borrowed from the level 0 one.
I’ll see what I can find.
Jos
… On 3 jul. 2017, at 21:20, Takahiro Ueda ***@***.***> wrote:
Here is another spam of valgrind log (vorm) with increasing SortIOSize:
FORM 4.1 (Jul 3 2017, v4.1-20131025-383-gd8ae7a6-dirty) 64-bits Run: Mon Jul 3 21:05:02 2017
#:SortIOSize 10M
#:MaxTermSize 600
#:SubTermsInSmall 16
#:SubLargePatches 1
#:SubFilePatches 2
S x;
CF f;
#define NTERMS "65"
L F = f(<x^1>+...+<x^`NTERMS'>);
P;
.end
StageSort
[PutIn: file->handle=3 file->POsize=298800]
[PutIn: file->handle=3 file->POsize=298800]
[PutIn: file->handle=3 file->POsize=298800]
[PutIn: file->handle=3 file->POsize=298800]
[PutIn: file->handle=4 file->POsize=10000000]
==5619== Syscall param read(buf) points to unaddressable byte(s)
==5619== at 0x32E12DB670: __read_nocancel (in /lib64/libc-2.12.so)
==5619== by 0x50E343: Uread (unixfile.c:134)
==5619== by 0x4FE08C: ReadFile (tools.c:1158)
==5619== by 0x42B0B6: FillInputGZIP (compress.c:694)
==5619== by 0x4DDD98: PutIn (sort.c:1260)
==5619== by 0x4E1AA0: MergePatches (sort.c:3656)
==5619== by 0x4E391C: EndSort (sort.c:1066)
==5619== by 0x4BC6EA: InFunction (proces.c:2057)
==5619== by 0x4B8007: Generator (proces.c:3759)
==5619== by 0x4B8463: Generator (proces.c:3931)
==5619== by 0x4B9BE3: Processor (proces.c:404)
==5619== by 0x43762D: DoExecute (execute.c:838)
==5619== Address 0x15469408 is 0 bytes after a block of size 5,100,488 alloc'd
==5619== at 0x4A05A57: malloc (vg_replace_malloc.c:299)
==5619== by 0x4FF227: Malloc1 (tools.c:2236)
==5619== by 0x4DB0B4: AllocSort (setfile.c:901)
==5619== by 0x4DDBF0: NewSort (sort.c:619)
==5619== by 0x4BC64C: InFunction (proces.c:2033)
==5619== by 0x4B8007: Generator (proces.c:3759)
==5619== by 0x4B8463: Generator (proces.c:3931)
==5619== by 0x4B9BE3: Processor (proces.c:404)
==5619== by 0x43762D: DoExecute (execute.c:838)
==5619== by 0x44DAAA: ExecModule (module.c:274)
==5619== by 0x4AFB2D: PreProcessor (pre.c:962)
==5619== by 0x4E8747: main (startup.c:1605)
==5619==
[PutIn: file->handle=4 file->POsize=10000000]
It is very strange that PutIn() is called with file->POsize=10000000, i.e., SortIOSize. Then ReadFile() overflows from the buffer allocated by AllocSort for the function argument (only 5100488 bytes, valgrind says). GDB shows it is allocated with AllocSort (LargeSize=4000000, SmallSize=500000, SmallEsize=800000, TermsInSmall=16, MaxPatches=1, MaxFpatches=4, IOsize=32768). For TFORM, instead of SortIOSize, the value calculated from AM.S0->file.POsize is used (see threads.c:541 <https://github.com/vermaseren/form/blob/d8ae7a65e86d13024332389730a5abe790be83e8/sources/threads.c#L541-L547>).
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub <#211 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFLxEtTYxpUTKU9_4ZYMnVwWj82FZLSgks5sKT7tgaJpZM4OKqxS>.
|
|
Actually, the problematic |
|
If you enable gzipdebug do you see the same pattern of events as in #95 (comment) ? |
|
Yes, it reads from the s4b.0 file which involves staged sorting.
Maybe (just maybe) there is just one setting for the staged sort because it
can only occur one at a time. But the writing and the other buffers belong
to the higher level sorting (functions/$’s). I am investigating those possibilities.
Jos
… On 3 jul. 2017, at 21:30, Takahiro Ueda ***@***.***> wrote:
Actually, the problematic PutIn() is called with AR.FoStage4[0]:
Breakpoint 1, PutIn ***@***.***=0x797d30 <A+7664>, position=0x7fff4e4ba520, buffer=0x7fff4e4bb8e0,
***@***.***=0x7fffffffcf40, ***@***.***=0) at sort.c:1242
(gdb) p &AR.FoStage4[0]
$14 = (FILEHANDLE *) 0x797d30 <A+7664>
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub <#211 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFLxEt9bvCCAQuJcLrk5MGQ-KM4fdqKQks5sKUFdgaJpZM4OKqxS>.
|
|
@jodavies No, I can see only |
|
OK. I didn't enable |
|
OK, I think I start understanding what happens.
At sorting level 0 we have big buffers and we have AR.FoStage4[] with those settings.
The settings for functions and dollars are considerably smaller. In the dollar FORM works
with those smaller settings, because it was not envisioned that you send functions and
dollars into stage4. This would be outrageously inefficient because each object like that
would need file accesses. Keep in mind that this was created originally just for functions.
Before version 3 the dollars did not exist.
Now, the first three stages in the sort for the functions/dollars take place with the smaller
settings, but now, when the stage4 enters, it looks like we get a funny mixture of buffer
sizes which can cause the reading to think that it needs to read more bytes than available
on the file (not really a problem) into a buffer that may be too small for all the stuff
requested. This can cause various types of problems. One can be a read error. Alternatively
there can be another buffer after this and if it reads a patch from the middle of the file
it will read too many bytes and overwrite this other buffer. The rest is mayhem.
This is how far I understand it at the moment. I have to think carefully how to give each
higher level a somewhat different set of buffers.
For the moment (good strategy anyway): Stay away from stage4 in dollars and functions.
Jos
… On 3 jul. 2017, at 21:30, Takahiro Ueda ***@***.***> wrote:
Actually, the problematic PutIn() is called with AR.FoStage4[0]:
Breakpoint 1, PutIn ***@***.***=0x797d30 <A+7664>, position=0x7fff4e4ba520, buffer=0x7fff4e4bb8e0,
***@***.***=0x7fffffffcf40, ***@***.***=0) at sort.c:1242
(gdb) p &AR.FoStage4[0]
$14 = (FILEHANDLE *) 0x797d30 <A+7664>
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub <#211 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFLxEt9bvCCAQuJcLrk5MGQ-KM4fdqKQks5sKUFdgaJpZM4OKqxS>.
|
Currently the test fails with asan/valgrind (#248). This will be fixed by the next commit.
|
With the current master branch a53fd8f, I locally (Ubuntu on Windows, Xenial, Homebrew gcc 5.5.0_4) have a TFORM problem: always leading to I didn't get the same error with previous versions. Confusingly, on another machine (CentOS 6.10, gcc Red Hat 7.3.1-5), this error occurs relatively rarely. The error did not occur on Travis CI: https://travis-ci.org/vermaseren/form/jobs/436825672, and it did fail on Circle CI once: https://circleci.com/gh/tueda/form/156. Does anyone also hit the same gzip error with the test |
|
Ah, maybe this is related to the fact that TFORM now consumes a crazy amount of memory even if it performs nothing (~ 8GB for |
|
The change of memory usage for doing nothing: FORM is now really heavy software in a sense of memory allocation. This may explain why the 32-bit test never succeeds (over the 4GB limit). |
- Update the Ruby lint configuration for RuboCop 0.59.2. (.rubocop.yml, check.rb) - Ignore Valgrind "set address range perms" warnings in the test suite (otherwise succeeded? returns false), which happen when FORM allocates a big memory chunk. (check.rb) - Fix a LaTeX typesetting error. (setup.tex) - Collect Jos's laptop settings into the usual place. (optimize.cc -> form3.h) - Suppress memory usage on 32-bit machines, especially for the 2^32 ~ 4G limit and tform -w4. (fsizes.h) - Don't include any system header files before config.h, because of the large file support on 32-bit machines. (gentopo.cc) Note that the Issue211 test may still fail. See the recent comments on Issue #211.
|
A valgrind dump: |
|
Another dump, the same problem and same environment, but different errors: |
Now the test suite script recognises Valgrind errors like "Syscall param ... points to unaddressable byte(s)" and makes warning? return true. The test of #211 fails for "valgrind tvorm" on Travis CI as it should do.
|
I modified the test suite script such that it can realises The "Conditional jump or move depends on uninitialised value(s)" of The "pointing to unaddressable bytes" looks a real and more serious buffer-overflow bug. |
One needs to revert these changes (to check if it succeeds) when the stage 4 sorting of function arguments and $-variables is correctly implemented in future (#211).
|
Reporting that I got an "Unexpected code in ReNumber" crash. I was running TFORM 5.0.0-beta.1 (Mar 15 2024, v5.0.0-beta.1-42-g2663e14) 127 workers. The crash occurred after a very long sequence of StageSorts within one module; an average of 440 for each of the 127 threads. Is this likely to be the same bug you guys were working on here, and if so what's a good strategy to avoid it? I guess I anyway need a much bigger LargePatches setting to get a sane runtime. Will this get rid of the crash as well? |
|
As I understand, it is hard to say: the Are you able to provide an example of your input and form script in this case? Is it working with a lot of large function arguments or dollar variables? These days form is supposed to terminate with an error, if a function arg or dollar sort reaches stage4 sorting. As a first tip, I would say that if you have any other parallelization opportunities (over diagrams, for eg) you should run more concurrent form jobs with fewer threads. The performance does not scale so well up to 128 threads (this statement does depend heavily on what exactly the form script does, though) and with fewer threads, each thread gets more memory space and you shouldn't stagesort as often or at all. Increasing your buffer sizes to avoid stagesorts will also help. |
|
If I'm not mistaken, the module in question is just one line: |
|
It seems like you are working at the "ground level" though, so the stagesorts are caused by a large number of generated terms, and your crash shouldn't be related to sub-buffers. Are you able to provide the expression just before this module, if it is not too big already, or at least a sample of the terms? |
|
I did not print any terms before the crash, but I do have a recovery file. Is that useful to you? |
|
I don't think so, without all of the scripts which come with it. Did it take a long time to arrive at this module (i.e., can you just add a print statement in the module before, and re-run) ? |
|
It did take a couple of hours to reach that point, and there are already ~50M terms before the module. But looking at the code, I think the arguments of |
|
Without seeing the whole terms I can't tell what the To continue investigating, I would add a print before the crashing module and simply re-run the code with the same threads/memory settings etc. Then you can see if you can reproduce the crash yourself, and at the same time we can see what the expression looks like exactly. |
|
Advise: Put a bracket statement in tenredgg.
… On 8 Apr 2024, at 17:25, jodavies ***@***.***> wrote:
Without seeing the whole terms I can't tell what the d_ are going to end up contracting with etc, so I it seems unlikely that your crash can be reproduced without more information.
To continue investigating, I would add a print before the crashing module and simply re-run the code with the same threads/memory settings etc. Then you can see if you can reproduce the crash yourself, and at the same time we can see what the expression looks like exactly.
—
Reply to this email directly, view it on GitHub <#211 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJPCEQMA72C4IYOOY64NMLY4KZFFAVCNFSM4DRKVRJKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBUGMYDIOBXHA4A>.
You are receiving this because you modified the open/close state.
|
|
|
It is for the investigation. I have two methods:
1: B tenredgg;
Print;
.sort
id tenredgg(?mu) = dd_(?mu);
2: This is for when the crash occurs before the final sort:
Print "%t”;
id tenredgg(?mu) = dd_(?mu);
This last one would give the offending term as the last term printed.
If tenredgg occurs more than once in some terms, the best would be to do just
one of them and then sort, and then one again etc.
… On 8 Apr 2024, at 17:56, ahamaline ***@***.***> wrote:
Advise: Put a bracket statement in tenredgg.
Can you explain more? Is this to help avoid the crash, or to investigate it?
—
Reply to this email directly, view it on GitHub <#211 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJPCEUSKGASOJU6P7ZFHJ3Y4K42BAVCNFSM4DRKVRJKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBUGMYTCOBTGA4Q>.
You are receiving this because you modified the open/close state.
|
|
OK, I'm running it now with the first of those options. |
|
So I got to the printout but not yet to the crash. Here is an example term: outside the brackets we have 'Mom[p2,nu2]' is really just a way to write p2[nu2], chosen specifically to avoid having the vector contract automatically. |
|
Ok, I found an example term with sixteen indices in tenredgg (in the printout before the module): The code has been running for 17.5 hours now and has neither crashed nor finished the module. |
|
If you resolve the I imagine this is not going to be easily reproducible. Possibly with a larger portion of your input expression and artificially small buffer settings or something like that. Do you have the log file from the crashed run? |
|
Another thing that may influence the running:
You have 1/(-1+D). This denominator treatment is notoriously weak and it is better to define your own
denominator function as in
CF Den;
and then use Den(-1+D).
That way it is also easier to have statements like
repeat id D*Den(-1+D) = 1+Den(-1+D);
or more general code as you can find in the second and third lecture in
https://www.nikhef.nl/~form/maindir/courses/uam2022/uam2022.html
Although not likely, it is not inconceivable that the error is related to this, because I cannot see what else you
do with these denominators.
… On 9 Apr 2024, at 14:23, jodavies ***@***.***> wrote:
If you resolve the Mom into actual scalar products where you have repeated indices you will get a much smaller result, but I don't know what the rest of your code does and it is not really relevant to the fact that you got the crash.
I imagine this is not going to be easily reproducible. Possibly with a larger portion of your input expression and artificially small buffer settings or something like that.
Do you have the log file from the crashed run?
—
Reply to this email directly, view it on GitHub <#211 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJPCEST3MGLWAMXBYUHBNDY4PMTPAVCNFSM4DRKVRJKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMBUGUYDIOJZGQZQ>.
You are receiving this because you modified the open/close state.
|
|
Yes, I agree that the code is absurdly inefficient for my use case. I did not write it and I don't understand the reasons for the authors' choices, so changing things is scary, but it looks like the savings would be worthwhile. The large number of terms is mainly due to the combinatorial explosion of index configurations, which would be avoided if we used either vectors or dummy indices. I did not save the log file; it did not have much that seemed informative. Just 56,000 StageSort reports from the various threads, followed by the error. Thank you for the tip about avoiding denominators. But how could the denominator be related to a crash that occurs in the I will add that the error must not be deterministic, since this run has already continued for many more StageSorts than the one that crashed. |
Hello,
I was writing a small script to see to what extent I could increase the test coverage in
sort.candcompress.c. I have something like the following:If one uncomments the
#: SubTermsInSmall 64setting,tformcrashes withUnexpected code in ReNumber.formdoes not crash, buttest2contains many f(...) terms that are not insideg.Thanks,
Josh
The text was updated successfully, but these errors were encountered: