ERL-44: bit_syntax and default signedness

[OTP-Maintainer]

Original reporter: nomorecoffee
Affected versions: R16B, R16A, R15B03-1, OTP-18.1, OTP-18.0, OTP-17.5, OTP-17.4, OTP-17.3, OTP-17.0, R16B03-1, R16B03, R16B02, R16B01
Fixed in version: OTP-19.0
Component: compiler
Migrated from: https://bugs.erlang.org/browse/ERL-44

---

The documentation said, ["The default signedness is unsigned"|http://www.erlang.org/doc/programming_examples/bit_syntax.html#Defaults] but in fact it doesn't work

{code}
1> << -1 >> = << 255 >>.
** exception error: no match of right hand side value <<"ÿ">>

3> << -1:8/unsigned >> = << 255 >>.
** exception error: no match of right hand side value <<"ÿ">>
{code}

{code}
4> A = << -1 >>.
<<"ÿ">>
5> A = << 255 >>.
<<"ÿ">>
{code}

{code}
6> << -1 >> == << 255 >>.
true
7> << -1 >> =:= << 255 >>.
true
{code}

You can test it *[here|http://tryerl.seriyps.ru/#id=2bf3]*.

{code:title=main.s}
{function, test_1, 0, 6}.
  {label,5}.
    {line,[{location,"main.erl",11}]}.
    {func_info,{atom,main},{atom,test_1},0}.
  {label,6}.
    {move,{literal,<<"ÿ">>},{x,0}}.
    {test,bs_start_match2,{f,7},1,[{x,0},0],{x,1}}.
    {test,bs_get_integer2,
          {f,7},
          2,
          [{x,1},
           {integer,8},
           1,
           {field_flags,[{anno,[12,{file,"main.erl"}]},unsigned,big]}],
          {x,2}}.
    {test,is_eq_exact,{f,7},[{x,2},{integer,-1}]}.
    {test,bs_test_tail2,{f,7},[{x,1},0]}.
    {move,{literal,<<"ÿ">>},{x,0}}.
    return.
  {label,7}.
    {line,[{location,"main.erl",12}]}.
    {badmatch,{x,0}}.
{code}

{code:title=main.core}
'test_1'/0 =
    %% Line 11
    fun () ->
	%% Line 12
	case #{#<255>(8,1,'integer',['unsigned'|['big']])}# of
	  <#{#<-1>(8,1,'integer',['unsigned'|['big']])}#> when 'true' ->
	      ( #{#<255>(8,1,'integer',['unsigned'|['big']])}#
		-| ['compiler_generated'] )
	  ( <_cor0> when 'true' ->
		primop 'match_fail'
		    ({'badmatch',_cor0})
	    -| ['compiler_generated'] )
	end
{code}

[OTP-Maintainer]

gomoripeti said:

While == and =:= ops return whether the left and right side expressions are equal (after they are evaluated) = is actually a pattern matching.

From main.S it is visible that what happens is first an unsigned integer is matched out of the right side binary then it is compared to -1.
(which is similar to writing the below where UnsignedInt =:= 255)

{code:erlang}
<< UnsignedInt >> = << 255 >>,
case UnsignedInt =:= -1 of
    true -> << 255 >>;
    _ -> error({badmatch, << 255 >>})
end
{code}


On the other hand if you explicitly declare signedness match can succeed

{code:erlang}
1> <<-1>> = <<255>>.
** exception error: no match of right hand side value <<"ÿ">>
2> <<-1/unsigned>> = <<255>>.
** exception error: no match of right hand side value <<"ÿ">>
3> <<-1/signed>> = <<255>>.
<<"ÿ">>
{code}

[OTP-Maintainer]

nomorecoffee said:

This correct behavior it is also possible to close a task?
{code}
1> <<-1>> = <<-1>>.
** exception error: no match of right hand side value <<"ÿ">>
{code}

[OTP-Maintainer]

nomorecoffee said:

It's look like a trap.

[OTP-Maintainer]

nomorecoffee said:

{code}
15> <<255:1>> = <<255:1>>.
** exception error: no match of right hand side value <<1:1>>
16> <<-1:1>> = <<255:1>>. 
** exception error: no match of right hand side value <<1:1>>
17> <<-1:1/signed>> = <<255:1>>.
<<1:1>>
18> <<1:1/signed>> = <<255:1>>. 
** exception error: no match of right hand side value <<1:1>>
19> <<1:1>> = <<255:1>>.       
<<1:1>>
{code}

[OTP-Maintainer]

gomoripeti said:

You brought up nice examples to show how this behaves. I try to add some more details.

So the = operator does pattern matching and on its two sides there are different "things". The right side is an expression (binary construction) which is evaluated before =. On the left side is a (binary) pattern. They behave differently.

When constructing a binary integers which cannot be stored on the given bytes are truncated or masked. For example 5 ( 2#101) storing on 2 bits only 1 is stored
{code}
1> <<5:2>>.
<<1:2>>
{code}

On the other hand (as discussed previously) the binary pattern works differently. It first matches out an integer from the right side binary based on the given bit size and signedess (8 bit unsigned integer by default) and then compares it to the literal integer that you provided in the pattern. As a result if you provide an integer in the pattern which is out of range of the given bits/signedness it will never match.
From your examples 255 can never match a 1 bit integer and -1 can never match an unsigned 8 bit integer:
{code}
2> <<255:1>> = Anything.
** exception error: no match of right hand side value ...
3> <<-1:8>> = Anything.
** exception error: no match of right hand side value ...
{code}

Hope this sheds some light for you on how binary pattern matching works. I understand that it can be a bit misleading at first that the same character sequence has different meaning depending on the context.

[OTP-Maintainer]

nomorecoffee said:

Thanks, [~gomoripeti]!

[OTP-Maintainer]

reith said:

Thanks for explanation.  It would nice to compiler throw a warning if size and signedness specification provided for a bound variable or literal, in matching phrase.

[OTP-Maintainer]

bjorn said:

In OTP 19, the compiler will generate warnings for clauses that will never match.