RabbitMQ binary files are inconsistent between different compiles

[EricLin89]

Here is a compilation issue of RabbitMQ, it's related to Elixir and mix, so I'm also posting it here to look for some help.

=====================================================================

We are using RabbitMQ in a commercial product, which delivers the installation package(with RabbitMQ included) to customer.
So it requires all the binary files in the package are consistent between different compiles, under a given release of source code.

We are compiling RabbitMQ 3.8.14 with erlang 23.0 and Elixir 1.10.4. The compilation process is run inside a chroot environment, which was built by running "yum install basesystem erlang elixir ... ", the yum repos are fixed.

We have a tool for comparing 2 compilation results, and it complains about some binary files are inconsistent. Those are:
Elixir.Collectable.beam
Elixir.CSV.Encode.beam
Elixir.Enumerable.beam
Elixir.IEx.Info.beam
Elixir.Inspect.beam
Elixir.JSON.Decoder.beam
Elixir.JSON.Encoder.beam
Elixir.List.Chars.beam
Elixir.RabbitMQ.CLI.Core.DataCoercion.beam
Elixir.String.Chars.beam
The above files are extracted from "rabbitmqctl" file.

I've made some digging into the makefiles, it seems the make process compiles something from elixir into rabbitmqctl using the "mix" command. But what happens inside "mix" is just beyond my capability.

Another point is this inconsistency only happens when the two compiles are executed on different machines, which is weired because the chroot environment should block the difference of hostos. Also actually the different machines are installed with the same host images.

Acually we have some tools to fix some variables(timestamps, randoms, etc) through the compilation process, but I just don't know where to "fix". Could anyone help to figure it out?

[josevalim]

Can you please call this on both machines?

iex(3)> Inspect.module_info()
[
  module: Inspect,
  exports: [
    __info__: 1,
    __protocol__: 1,
    impl_for: 1,
    impl_for!: 1,
    inspect: 2,
    behaviour_info: 1,
    module_info: 0,
    module_info: 1
  ],
  attributes: [
    vsn: [209714863393261072825932680805028164758],
    dialyzer: [nowarn_function: [__protocol__: 1, impl_for: 1, impl_for!: 1]],
    __protocol__: [fallback_to_any: true]
  ],
  compile: [
    version: '8.0',
    options: [
      :no_spawn_compiler_process,
      :from_core,
      :no_core_prepare,
      :no_auto_import,
      {:inline, [struct_impl_for: 1]}
    ],
    source: '/Users/jose/OSS/elixir/lib/elixir/lib/inspect.ex'
  ],
  md5: <<157, 197, 157, 15, 152, 104, 54, 212, 199, 245, 207, 68, 67, 203, 116,
    150>>
]

Do this for both Inspect and RabbitMQ.CLI.Core.DataCoercion please. Those all seem to be protocols, which help us narrow down the root cause.

[EricLin89]

Thanks, I found those .beam files in the _build/dev/lib/rabbitmqctl/consolidated directory, and I called the given code under this directory. I'm not sure if I did it right, but I'll still post the output here.

machine 1:

bash-5.0# iex
Erlang/OTP 23 [erts-11.0] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [hipe]

Interactive Elixir (1.10.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Inspect.module_info()
[
  module: Inspect,
  exports: [
    __info__: 1,
    __protocol__: 1,
    impl_for: 1,
    impl_for!: 1,
    inspect: 2,
    behaviour_info: 1,
    module_info: 0,
    module_info: 1
  ],
  attributes: [
    vsn: [275458473139948013867881782065428346862],
    dialyzer: [nowarn_function: [__protocol__: 1, impl_for: 1, impl_for!: 1]],
    protocol: [fallback_to_any: true]
  ],
  compile: [
    version: '7.3.2',
    options: [
      :dialyzer,
      :no_spawn_compiler_process,
      :from_core,
      :no_core_prepare,
      :no_auto_import,
      {:inline, [struct_impl_for: 1]}
    ],
    source: '/root/rpmbuild/BUILD/elixir-1.10.4/lib/elixir/lib/inspect.ex'
  ],
  native: false,
  md5: <<207, 59, 95, 71, 141, 55, 235, 46, 217, 190, 34, 136, 124, 246, 183,
    238>>
]
iex(2)> RabbitMQ.CLI.Core.DataCoercion.module_info()
[
  module: RabbitMQ.CLI.Core.DataCoercion,
  exports: [
    __info__: 1,
    impl_for!: 1,
    to_atom: 1,
    impl_for: 1,
    __protocol__: 1,
    behaviour_info: 1,
    module_info: 0,
    module_info: 1
  ],
  attributes: [
    vsn: [113828052217736202921823743469781253720],
    dialyzer: [nowarn_function: [__protocol__: 1, impl_for: 1, impl_for!: 1]],
    protocol: [fallback_to_any: false]
  ],
  compile: [
    version: '7.6',
    options: [
      :dialyzer,
      :no_spawn_compiler_process,
      :from_core,
      :no_core_prepare,
      :no_auto_import,
      {:inline, [struct_impl_for: 1]}
    ],
    source: '/builddir/build/BUILD/rabbitmq-3.8.14/deps/rabbitmq_cli/lib/rabbitmq/cli/core/data_coercion.ex'
  ],
  native: false,
  md5: <<85, 162, 124, 68, 254, 17, 13, 121, 217, 208, 205, 149, 175, 95, 254,
    88>>
]

machine 2:

bash-5.0# iex
Erlang/OTP 23 [erts-11.0] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:1] [hipe]

Interactive Elixir (1.10.4) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Inspect.module_info()
[
  module: Inspect,
  exports: [
    __info__: 1,
    __protocol__: 1,
    impl_for: 1,
    impl_for!: 1,
    inspect: 2,
    behaviour_info: 1,
    module_info: 0,
    module_info: 1
  ],
  attributes: [
    vsn: [275458473139948013867881782065428346862],
    dialyzer: [nowarn_function: [__protocol__: 1, impl_for: 1, impl_for!: 1]],
    protocol: [fallback_to_any: true]
  ],
  compile: [
    version: '7.3.2',
    options: [
      :dialyzer,
      :no_spawn_compiler_process,
      :from_core,
      :no_core_prepare,
      :no_auto_import,
      {:inline, [struct_impl_for: 1]}
    ],
    source: '/root/rpmbuild/BUILD/elixir-1.10.4/lib/elixir/lib/inspect.ex'
  ],
  native: false,
  md5: <<207, 59, 95, 71, 141, 55, 235, 46, 217, 190, 34, 136, 124, 246, 183,
    238>>
]
iex(2)> RabbitMQ.CLI.Core.DataCoercion.module_info()
[
  module: RabbitMQ.CLI.Core.DataCoercion,
  exports: [
    __info__: 1,
    impl_for!: 1,
    to_atom: 1,
    impl_for: 1,
    __protocol__: 1,
    behaviour_info: 1,
    module_info: 0,
    module_info: 1
  ],
  attributes: [
    vsn: [26859511837667837699412160852075174438],
    dialyzer: [nowarn_function: [__protocol__: 1, impl_for: 1, impl_for!: 1]],
    protocol: [fallback_to_any: false]
  ],
  compile: [
    version: '7.6',
    options: [
      :dialyzer,
      :no_spawn_compiler_process,
      :from_core,
      :no_core_prepare,
      :no_auto_import,
      {:inline, [struct_impl_for: 1]}
    ],
    source: '/builddir/build/BUILD/rabbitmq-3.8.14/deps/rabbitmq_cli/lib/rabbitmq/cli/core/data_coercion.ex'
  ],
  native: false,
  md5: <<20, 52, 244, 45, 39, 37, 97, 73, 9, 21, 13, 160, 44, 111, 90, 38>>
]

[josevalim]

@EricLin89 that's exactly what I needed but unfortunately the change between .beam files are not superficial, so I will need to byte compare them. Could you zip all 4 beam files above and either attach them here (not sure if possible) or send them to my email (you can find it in my profile)? I will debug them when I can and follow up. Thanks.

[EricLin89]

Due to the information security policy of the company, I'll have to go through some application process. I've already applied to send those files to your email, but I'm not sure when or by which form you'll receive them. Please leave me a message if you have problems retriving them.

[josevalim]

@EricLin89 sounds good! I will at least ack receiving them within 24 hours. If you don't hear from me, then ping me, because it means something got lost in the process. Have a good weekend.

[josevalim]

I got the files and I was able to spot the difference in the bytecode, thank you! A fix is coming shortly!

[EricLin89]

Great! I'll give it a try

[EricLin89]

Thanks, it works!