Segmentation fault on sunsparc64 platform in HASH_FUNCTION

[GoogleCodeExporter]

On sunsparc64 machine, the function fastlz2_compress() for FastLZ level 2
results in SEGV while accessing invalid memory pointed by ip in
HASH_FUNCTION at line 376. 

This happens only when the input data pattern encounters 5-byte match as
given below and distance is greater than 1 indicating it's not a run. The
for loop given below makes an assumption that the ip_limit condition
restricts ip to be atleast 8 less than ip_bound value. But when we find
5-byte match, the ip value may exceed ip_bound value. This results in
accessing invalid memory within HASH_FUNCTION.

Original issue reported on code.google.com by aditya...@gmail.com on 2 Feb 2010 at 10:30

[GoogleCodeExporter]

The likely fix would be to change the assumption made for ip value in this for 
loop
and allow only 6 increments as safe condition for FASTLZ_LEVEL 2. The 
assumption of 8
increments is safe only for FASTLZ_LEVEL 1.

275     for(;;)
276     {
277       /* safe because the outer check against ip limit */
278       if(*ref++ != *ip++) break;
279       if(*ref++ != *ip++) break;
280       if(*ref++ != *ip++) break;
281       if(*ref++ != *ip++) break;
282       if(*ref++ != *ip++) break;
283       if(*ref++ != *ip++) break;
284 #if FASTLZ_LEVEL==1
285       if(*ref++ != *ip++) break;
286       if(*ref++ != *ip++) break;
287 #endif
288       while(ip < ip_bound)
289         if(*ref++ != *ip++) break;
290       break;
291     }

Original comment by aditya...@gmail.com on 2 Feb 2010 at 10:31