Fixed undefined behavior in netmask generation #1717

[rfrohl]

Tested this change on x86_64, i586, ppc, ppc64, ppc64le, aarch64 and s390x. Unit tests passed successfully on these architectures.

Also played around with the test binary (addrset) a bit and could not find any obvious problems with the change.

[nnposter]

Over at #1717 I have proposed an alternate patch, which also disposes of the rather complex arithmetic in the original code.

[nnposter]

Over at #1717 I have proposed an alternate patch, which also disposes of the rather complex arithmetic in the original code.

[nnposter]

This code suffers from a similar issue as the original one: There is no guaranteed outcome when unmasked_bits exceeds 31. As an example:

#include <stdio.h>
void main ()
{
int m=32;
printf("%08x\n%08x\n",(0xFFFFFFFF<<(m-1)<<1),(0xFFFFFFFF<<m));
}

When compiled with GCC on Ubuntu 18.04 x64:

$ ./a.out 
00000000
ffffffff

[rfrohl]

I assumed that it would hit another branch if unmasked_bits is a multiple of 32. But I my understanding of how mask is used in the rest of the code is limited, so I might miss something here.

Will test the other patch from the ticket on Monday