New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

swf output not compatible with flash player 27 #6589

Closed
andyli opened this Issue Sep 17, 2017 · 25 comments

Comments

Projects
None yet
2 participants
@andyli
Member

andyli commented Sep 17, 2017

Tested with Haxe 3.4.3 and a0577b7.

A hello world program:

class HelloWorld {
	static function main():Void {
		trace("Hello world!");
	}
}

Compiled with

-main HelloWorld
-cp src
-swf bin/HelloWorld.swf

Run with Flash Player Debugger 27.0.0.130 (on a Mac)

Warning: Failed to parse corrupt data.
VerifyError: Error #1001: The method MethodInfo-49() is not implemented.

        at HelloWorld$/main()
        at boot_1ee6/init()
        at flash::Boot/start()
        at boot_1ee6()

If run with Flash Player (not Debugger), the Flash Player will simply crash.
Quick link for FP download: https://www.adobe.com/support/flashplayer/debug_downloads.html

@Simn

This comment has been minimized.

Show comment
Hide comment
@Simn

Simn Sep 17, 2017

Member

Argh, so that was a real issue after all... It came up on our CI, but I was assuming that it's just some stupid setup issue because it only happened on OS X.

No idea right now what could be causing this...

Member

Simn commented Sep 17, 2017

Argh, so that was a real issue after all... It came up on our CI, but I was assuming that it's just some stupid setup issue because it only happened on OS X.

No idea right now what could be causing this...

@Simn

This comment has been minimized.

Show comment
Hide comment
@Simn

Simn Sep 17, 2017

Member

Could you check if this is a regression from 3.4.2? There are no changes to the SWF generator between 3.4.2 and 3.4.3, so I'm very puzzled about this.

Member

Simn commented Sep 17, 2017

Could you check if this is a regression from 3.4.2? There are no changes to the SWF generator between 3.4.2 and 3.4.3, so I'm very puzzled about this.

@andyli

This comment has been minimized.

Show comment
Hide comment
@andyli

andyli Sep 17, 2017

Member

Just checked. The swf produced with Haxe 3.4.2 runs fine...

Member

andyli commented Sep 17, 2017

Just checked. The swf produced with Haxe 3.4.2 runs fine...

@andyli

This comment has been minimized.

Show comment
Hide comment
@andyli

andyli Sep 17, 2017

Member

Looks like it's an OCaml issue...

  • Haxe 3.4.3 compiled with OCaml 4.02.3 didn't reproduce this issue.
  • Haxe 3.4.3 compiled with OCaml 4.05.0 reproduced this issue.
Member

andyli commented Sep 17, 2017

Looks like it's an OCaml issue...

  • Haxe 3.4.3 compiled with OCaml 4.02.3 didn't reproduce this issue.
  • Haxe 3.4.3 compiled with OCaml 4.05.0 reproduced this issue.
@Simn

This comment has been minimized.

Show comment
Hide comment
@Simn

Simn Sep 17, 2017

Member

Whaaaat...

Member

Simn commented Sep 17, 2017

Whaaaat...

andyli added a commit to andyli/haxe that referenced this issue Sep 17, 2017

@andyli

This comment has been minimized.

Show comment
Hide comment
@andyli

andyli Sep 17, 2017

Member

I've just pushed to the three_four_three branch to use OCaml 4.02.3 to build Haxe on Mac, and replaced the 3.4.3 Mac binaries and installer to use the new build artifacts.

We still need to investigate OCaml 4.05's issue since we can't use 4.02.3 forever.

Member

andyli commented Sep 17, 2017

I've just pushed to the three_four_three branch to use OCaml 4.02.3 to build Haxe on Mac, and replaced the 3.4.3 Mac binaries and installer to use the new build artifacts.

We still need to investigate OCaml 4.05's issue since we can't use 4.02.3 forever.

@Simn

This comment has been minimized.

Show comment
Hide comment
@Simn

Simn Sep 18, 2017

Member

What I don't understand (besides everything else) is why it only fails on 3.4.3 and not on development.

Member

Simn commented Sep 18, 2017

What I don't understand (besides everything else) is why it only fails on 3.4.3 and not on development.

@andyli

This comment has been minimized.

Show comment
Hide comment
@andyli

andyli Sep 18, 2017

Member
Member

andyli commented Sep 18, 2017

@Simn

This comment has been minimized.

Show comment
Hide comment
@Simn

Simn Sep 18, 2017

Member

But our CI doesn't fail on development, whereas it was failing on the 3.4.3 branch. Where's the difference?

Member

Simn commented Sep 18, 2017

But our CI doesn't fail on development, whereas it was failing on the 3.4.3 branch. Where's the difference?

@andyli

This comment has been minimized.

Show comment
Hide comment
@andyli

andyli Sep 18, 2017

Member
Member

andyli commented Sep 18, 2017

@Simn

This comment has been minimized.

Show comment
Hide comment
@Simn

Simn Sep 18, 2017

Member

It was failing: https://travis-ci.org/HaxeFoundation/haxe/jobs/274558244#L4379

I disabled the tests because I thought it was some setup issue, and I really needed the builds before the conference.

Member

Simn commented Sep 18, 2017

It was failing: https://travis-ci.org/HaxeFoundation/haxe/jobs/274558244#L4379

I disabled the tests because I thought it was some setup issue, and I really needed the builds before the conference.

@andyli

This comment has been minimized.

Show comment
Hide comment
@andyli

andyli Sep 27, 2017

Member

Some more testing:

  • Haxe 3.4.3 compiled with OCaml 4.02.3 to 4.04.2 (inclusive) didn't reproduce this issue. The produced HelloWorld.swf files are exactly the same byte by byte according to cmp.
  • Haxe 3.4.3 compiled with OCaml 4.05.0 reproduced this issue. The HelloWorld.swf file produced is largely different from the correct one.

The hex dump of the two swf files:
HelloWorld_4.04.2.swf.hex.txt
HelloWorld_4.05.0.swf.hex.txt

Member

andyli commented Sep 27, 2017

Some more testing:

  • Haxe 3.4.3 compiled with OCaml 4.02.3 to 4.04.2 (inclusive) didn't reproduce this issue. The produced HelloWorld.swf files are exactly the same byte by byte according to cmp.
  • Haxe 3.4.3 compiled with OCaml 4.05.0 reproduced this issue. The HelloWorld.swf file produced is largely different from the correct one.

The hex dump of the two swf files:
HelloWorld_4.04.2.swf.hex.txt
HelloWorld_4.05.0.swf.hex.txt

@andyli

This comment has been minimized.

Show comment
Hide comment
@andyli

andyli Sep 27, 2017

Member

The current development branch passes because flash target is not tested in mac builds since b4221c3.

Member

andyli commented Sep 27, 2017

The current development branch passes because flash target is not tested in mac builds since b4221c3.

@Simn

This comment has been minimized.

Show comment
Hide comment
@Simn

Simn Sep 27, 2017

Member

Haha... I'm so stupid. x)

Member

Simn commented Sep 27, 2017

Haha... I'm so stupid. x)

@andyli

This comment has been minimized.

Show comment
Hide comment
@andyli

andyli Sep 29, 2017

Member

Turn out the issue isn't Mac specific - I successfully reproduced it on both Windows and Linux. It's just that we have been using OCaml 4.02.3 in CIs (except for Mac, in which the homebrew ocaml package is 4.05.0).

It also means that this is an serious issue because there is no way for the Linux package to build with an older OCaml version - they will just use the one available in the Linux distro. i.e. The flash target is broken in most recent Linux distro releases.

@ncannasse @Simn Would you try to find out what's happen here?

Quick reminder on how to use OCaml 4.05.0 to compile Haxe 3.4.3 in Windows:

# in a cygwin terminal
opam switch 4.05.0+mingw32c
eval `opam config env`
opam install camlp4
cd path/to/haxe
# clean everything and checkout 3.4.3
make clean && git clean -fdx && git reset --hard HEAD
git checkout 3.4.3
git submodule update --force
# compile
make
Member

andyli commented Sep 29, 2017

Turn out the issue isn't Mac specific - I successfully reproduced it on both Windows and Linux. It's just that we have been using OCaml 4.02.3 in CIs (except for Mac, in which the homebrew ocaml package is 4.05.0).

It also means that this is an serious issue because there is no way for the Linux package to build with an older OCaml version - they will just use the one available in the Linux distro. i.e. The flash target is broken in most recent Linux distro releases.

@ncannasse @Simn Would you try to find out what's happen here?

Quick reminder on how to use OCaml 4.05.0 to compile Haxe 3.4.3 in Windows:

# in a cygwin terminal
opam switch 4.05.0+mingw32c
eval `opam config env`
opam install camlp4
cd path/to/haxe
# clean everything and checkout 3.4.3
make clean && git clean -fdx && git reset --hard HEAD
git checkout 3.4.3
git submodule update --force
# compile
make
@Simn

This comment has been minimized.

Show comment
Hide comment
@Simn

Simn Sep 29, 2017

Member

Oh man... that could be some 10 year old Obj.magic coming back to haunt us.

Member

Simn commented Sep 29, 2017

Oh man... that could be some 10 year old Obj.magic coming back to haunt us.

@Simn

This comment has been minimized.

Show comment
Hide comment
@Simn

Simn Sep 29, 2017

Member

Compare -D dump output

Identical

This is good because it means the rest of the compiler works fine. This is bad because it means the problem is the SWF generator itself.

Compare in decompiler

header: identical (except file size)
frames: identical
others: identical
scripts: structure is identical:

scripts
  flash
    Boot
    Lib
  haxe
    Log
  Dynamic
  Main
  Std
  boot_54a1

Compare scripts

Boot: identical
Lib: identical
Log: different

4.02:

package haxe
{
   public class Log
   {
      
      public static function trace(v:*, infos:Object = undefined):void
      {
         Boot.__trace(v,infos);
      } 
      public function Log()
      {
      }
   }
}

4.05:

package haxe
{
   public class Log
   {
      
      public static function trace(param1:*, param2:Object = undefined):void
      {
      } 
      public function Log()
      {
      }
   }
}

Main: identical
Std: identical
boot_54a1: different

4.02:

package
{
   import flash.Boot;
   import flash.Lib;
   import flash.display.MovieClip;
   
   public dynamic class boot_54a1 extends Boot
   {
       
      public function boot_54a1()
      {
         super();
         if(Lib.current == null)
         {
            Lib.current = this;
         }
         start();
      }
      
      override public function init() : void
      {
         Math.NaN = Number.NaN;
         Math.NEGATIVE_INFINITY = Number.NEGATIVE_INFINITY;
         Math.POSITIVE_INFINITY = Number.POSITIVE_INFINITY;
         Math.isFinite = function(i:Number):Boolean
         {
            return isFinite(i);
         };
         Math.isNaN = function(i1:Number):Boolean
         {
            return isNaN(i1);
         };
         Main.main();
      }
   }
}

4.05:

package
{
   import flash.Boot;
   import flash.Lib;
   import flash.display.MovieClip;
   
   public dynamic class boot_54a1 extends Boot
   {
       
      public function boot_54a1()
      {
         super();
         if(Lib.current == null)
         {
            Lib.current = this;
         }
         start();
      }
      
      override public function init() : void
      {
         Math.NaN = Number.NaN;
         Math.NEGATIVE_INFINITY = Number.NEGATIVE_INFINITY;
         Math.POSITIVE_INFINITY = Number.POSITIVE_INFINITY;
         Math.isFinite = function(param1:Number):Boolean
         {
         };
         Math.isNaN = function(param1:Number):Boolean
         {
         };
         Main.main();
      }
   }
}

First thoughts

Some problem with function argument names, but why...

Member

Simn commented Sep 29, 2017

Compare -D dump output

Identical

This is good because it means the rest of the compiler works fine. This is bad because it means the problem is the SWF generator itself.

Compare in decompiler

header: identical (except file size)
frames: identical
others: identical
scripts: structure is identical:

scripts
  flash
    Boot
    Lib
  haxe
    Log
  Dynamic
  Main
  Std
  boot_54a1

Compare scripts

Boot: identical
Lib: identical
Log: different

4.02:

package haxe
{
   public class Log
   {
      
      public static function trace(v:*, infos:Object = undefined):void
      {
         Boot.__trace(v,infos);
      } 
      public function Log()
      {
      }
   }
}

4.05:

package haxe
{
   public class Log
   {
      
      public static function trace(param1:*, param2:Object = undefined):void
      {
      } 
      public function Log()
      {
      }
   }
}

Main: identical
Std: identical
boot_54a1: different

4.02:

package
{
   import flash.Boot;
   import flash.Lib;
   import flash.display.MovieClip;
   
   public dynamic class boot_54a1 extends Boot
   {
       
      public function boot_54a1()
      {
         super();
         if(Lib.current == null)
         {
            Lib.current = this;
         }
         start();
      }
      
      override public function init() : void
      {
         Math.NaN = Number.NaN;
         Math.NEGATIVE_INFINITY = Number.NEGATIVE_INFINITY;
         Math.POSITIVE_INFINITY = Number.POSITIVE_INFINITY;
         Math.isFinite = function(i:Number):Boolean
         {
            return isFinite(i);
         };
         Math.isNaN = function(i1:Number):Boolean
         {
            return isNaN(i1);
         };
         Main.main();
      }
   }
}

4.05:

package
{
   import flash.Boot;
   import flash.Lib;
   import flash.display.MovieClip;
   
   public dynamic class boot_54a1 extends Boot
   {
       
      public function boot_54a1()
      {
         super();
         if(Lib.current == null)
         {
            Lib.current = this;
         }
         start();
      }
      
      override public function init() : void
      {
         Math.NaN = Number.NaN;
         Math.NEGATIVE_INFINITY = Number.NEGATIVE_INFINITY;
         Math.POSITIVE_INFINITY = Number.POSITIVE_INFINITY;
         Math.isFinite = function(param1:Number):Boolean
         {
         };
         Math.isNaN = function(param1:Number):Boolean
         {
         };
         Main.main();
      }
   }
}

First thoughts

Some problem with function argument names, but why...

@Simn

This comment has been minimized.

Show comment
Hide comment
@Simn

Simn Sep 29, 2017

Member

Update: It's not the MultiArray implementation. I reverted that change back to Array/DynArray and still get the same behavior.

Member

Simn commented Sep 29, 2017

Update: It's not the MultiArray implementation. I reverted that change back to Array/DynArray and still get the same behavior.

@Simn

This comment has been minimized.

Show comment
Hide comment
@Simn

Simn Sep 29, 2017

Member

Someone on #ocaml mentioned that they had an issue with changed evaluation order which they addressed in this commit: facebook/infer@5b3c2c0

I have confirmed that a bytecode Haxe compiled from 4.04.2 also reproduces the issue, so it's almost definitely an evaluation order problem. Now I just have to find out where it comes from...

Member

Simn commented Sep 29, 2017

Someone on #ocaml mentioned that they had an issue with changed evaluation order which they addressed in this commit: facebook/infer@5b3c2c0

I have confirmed that a bytecode Haxe compiled from 4.04.2 also reproduces the issue, so it's almost definitely an evaluation order problem. Now I just have to find out where it comes from...

@Simn

This comment has been minimized.

Show comment
Hide comment
@Simn

Simn Sep 29, 2017

Member

Tag IDs + length:

Working:

69: 6
9: 5
86: 16
72: 6119
76: 16
1: 2

Broken:

69: 6
9: 5
86: 16
72: 5436
76: 16
1: 2

72 is TActionScript3, so it's what genswf9 produces.

Member

Simn commented Sep 29, 2017

Tag IDs + length:

Working:

69: 6
9: 5
86: 16
72: 6119
76: 16
1: 2

Broken:

69: 6
9: 5
86: 16
72: 5436
76: 16
1: 2

72 is TActionScript3, so it's what genswf9 produces.

@Simn

This comment has been minimized.

Show comment
Hide comment
@Simn

Simn Sep 29, 2017

Member

I finally found it and fixed it in HaxeFoundation/ocamllibs@b2d729b

Took me 5 hours to track this down... Please confirm that it works now.

Member

Simn commented Sep 29, 2017

I finally found it and fixed it in HaxeFoundation/ocamllibs@b2d729b

Took me 5 hours to track this down... Please confirm that it works now.

@andyli

This comment has been minimized.

Show comment
Hide comment
@andyli

andyli Sep 29, 2017

Member

Took me 5 hours to track this down...

I thought it could take even longer given how tricky it is, I'm so glad you fixed it in just 5 hours :)
I confirm the development branch compiles and works nicely using OCaml 4.05.

I think we need a 3.4.4 release soonish.

Member

andyli commented Sep 29, 2017

Took me 5 hours to track this down...

I thought it could take even longer given how tricky it is, I'm so glad you fixed it in just 5 hours :)
I confirm the development branch compiles and works nicely using OCaml 4.05.

I think we need a 3.4.4 release soonish.

@Simn

This comment has been minimized.

Show comment
Hide comment
@Simn

Simn Sep 29, 2017

Member

I think we need a 3.4.4 release soonish.

Agreed... can you start a branch? I made a bit of a mess in the 3.4.3 branch when I was in a hurry to get the release out.

Maybe we should try to get the RunCi changes onto that branch too so it's a bit cleaner. There was also a hardcoded "three_four_three" somewhere which shouldn't be there.

Member

Simn commented Sep 29, 2017

I think we need a 3.4.4 release soonish.

Agreed... can you start a branch? I made a bit of a mess in the 3.4.3 branch when I was in a hurry to get the release out.

Maybe we should try to get the RunCi changes onto that branch too so it's a bit cleaner. There was also a hardcoded "three_four_three" somewhere which shouldn't be there.

@andyli

This comment has been minimized.

Show comment
Hide comment
@andyli

andyli Sep 29, 2017

Member

No problem, I will sort this out in this weekend. Getting the RunCi split may be a bit too complex for the 3.4 branch (e.g. php/php7 change), but I will see what I can do there.

Member

andyli commented Sep 29, 2017

No problem, I will sort this out in this weekend. Getting the RunCi split may be a bit too complex for the 3.4 branch (e.g. php/php7 change), but I will see what I can do there.

andyli added a commit to andyli/haxe that referenced this issue Oct 3, 2017

andyli added a commit that referenced this issue Oct 3, 2017

@andyli

This comment has been minimized.

Show comment
Hide comment
@andyli

andyli Oct 5, 2017

Member

Closing this since the fix was applied to both development and 3.4_bugfix.

Member

andyli commented Oct 5, 2017

Closing this since the fix was applied to both development and 3.4_bugfix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment