Examples on Apple M1 need an additional resolution in esy. #160

pm-mck · 2021-09-23T19:57:20Z

As of today, I found that running the examples in esy on an Apple M1 machine requires this resolution:

  "resolutions": {
    "esy-openssl": "esy-packages/esy-openssl#619ae2d46ca981ec26ab3287487ad98b157a01d1"
  }

documented on the esy-openssl project page:

https://github.com/esy-packages/esy-openssl

If you don't do this, the esy build emits a few clang errors which will be difficult for a newcomer to handle.

Should this be added to the examples README or documented / provided in some other manner? Happy to open a pr but don't want to put it in the wrong place.

aantron · 2021-09-24T10:33:37Z

Can you make a PR that adds this to the resolutions of example/2-middleware here:

dream/example/2-middleware/esy.json

Lines 11 to 13 in 52d4da2

    
           "resolutions": { 
        
             "@opam/conf-libev": "esy-packages/libev:package.json#0b5eb6685b688649045aceac55dc559f6f21b829" 
        
           },

From your message, I suppose this will fix the example for you, and choosing this example for the PR will also get the change tested in CI, since that example is part of Dream's quick start script that gets run in CI. If everything works, I'll do the busywork of adding this resolution to all the other examples.

@ManasJayanth is it correct to use the resolution this way? I assume that esy-openssl#619ae2d46ca981ec26ab3287487ad98b157a01d1 will work on all platforms.

sagarjs · 2021-09-25T10:46:14Z

thank you for creating this @pm-mck
I was just about to create this myself though I have no idea how to fix the issue.

I tried adding the line to my esy.json in 2-middleware but I am getting the following error while running 'npx esy'
It looks like permission errors, but if i try manually creating the directory dynlinks as the logs seems to suggest, I do not get an error
what am I getting wrong here?

About the OCAML core installation:
        Standard library:      /Users/sagarsodah/.esy/3_____________________________________________________________/i/ocaml-4.12.0-44d44425/lib/ocaml
        Binaries:              /Users/sagarsodah/.esy/3_____________________________________________________________/i/ocaml-4.12.0-44d44425/bin
        Manual pages:          /Users/sagarsodah/.esy/3_____________________________________________________________/i/ocaml-4.12.0-44d44425/man
        Multi-threading type:  posix
    The directory of site-specific packages will be
        site-lib:              /Users/sagarsodah/.esy/3_____________________________________________________________/i/opam__s__ocaml_secondary_compiler-opam__c__4.08.1-1-eca62758/share/ocaml-secondary-compiler/lib
    The configuration file is written to:
        findlib config file:   /Users/sagarsodah/.esy/3_____________________________________________________________/i/ocaml-4.12.0-44d44425/etc/findlib.conf
    Software will be installed:
        Libraries:             in <site-lib>/findlib
        Binaries:              /Users/sagarsodah/.esy/3_____________________________________________________________/i/ocaml-4.12.0-44d44425/bin
        Manual pages:          /Users/sagarsodah/.esy/3_____________________________________________________________/i/ocaml-4.12.0-44d44425/man
        topfind script:        /Users/sagarsodah/.esy/3_____________________________________________________________/i/ocaml-4.12.0-44d44425/lib/ocaml
    Topfind ppxopt support:    yes
    Toolbox:                   no
    Link custom runtime:       yes
    Need bytes compatibility:  no
    
    Configuration has been written to Makefile.config
    
    You can now do 'make all', and optionally 'make opt', to build ocamlfind.
    # esy-build-package: running: 'make' 'install-meta'
    for x in dynlink stdlib unix bigarray compiler-libs ocamldoc threads str; do mkdir -p "/Users/sagarsodah/.esy/3_____________________________________________________________/i/opam__s__ocaml_secondary_compiler-opam__c__4.08.1-1-eca62758/share/ocaml-secondary-compiler/lib/$x"; cp site-lib-src/$x/META "/Users/sagarsodah/.esy/3_____________________________________________________________/i/opam__s__ocaml_secondary_compiler-opam__c__4.08.1-1-eca62758/share/ocaml-secondary-compiler/lib/$x"; done
    mkdir: /Users/sagarsodah/.esy/3_____________________________________________________________/i/opam__s__ocaml_secondary_compiler-opam__c__4.08.1-1-eca62758/share/ocaml-secondary-compiler/lib/dynlink: Operation not permitted
    cp: /Users/sagarsodah/.esy/3_____________________________________________________________/i/opam__s__ocaml_secondary_compiler-opam__c__4.08.1-1-eca62758/share/ocaml-secondary-compiler/lib/dynlink: Operation not permitted
    mkdir: /Users/sagarsodah/.esy/3_____________________________________________________________/i/opam__s__ocaml_secondary_compiler-opam__c__4.08.1-1-eca62758/share/ocaml-secondary-compiler/lib/stdlib: Operation not permitted
    cp: /Users/sagarsodah/.esy/3_____________________________________________________________/i/opam__s__ocaml_secondary_compiler-opam__c__4.08.1-1-eca62758/share/ocaml-secondary-compiler/lib/stdlib: Operation not permitted
    mkdir: /Users/sagarsodah/.esy/3_____________________________________________________________/i/opam__s__ocaml_secondary_compiler-opam__c__4.08.1-1-eca62758/share/ocaml-secondary-compiler/lib/unix: Operation not permitted
    cp: /Users/sagarsodah/.esy/3_____________________________________________________________/i/opam__s__ocaml_secondary_compiler-opam__c__4.08.1-1-eca62758/share/ocaml-secondary-compiler/lib/unix: Operation not permitted
    mkdir: /Users/sagarsodah/.esy/3_____________________________________________________________/i/opam__s__ocaml_secondary_compiler-opam__c__4.08.1-1-eca62758/share/ocaml-secondary-compiler/lib/bigarray: Operation not permitted
    cp: /Users/sagarsodah/.esy/3_____________________________________________________________/i/opam__s__ocaml_secondary_compiler-opam__c__4.08.1-1-eca62758/share/ocaml-secondary-compiler/lib/bigarray: Operation not permitted
    cp: /Users/sagarsodah/.esy/3_____________________________________________________________/i/opam__s__ocaml_secondary_compiler-opam__c__4.08.1-1-eca62758/share/ocaml-secondary-compiler/lib/compiler-libs/META: Operation not permitted
    cp: /Users/sagarsodah/.esy/3_____________________________________________________________/i/opam__s__ocaml_secondary_compiler-opam__c__4.08.1-1-eca62758/share/ocaml-secondary-compiler/lib/ocamldoc/META: Operation not permitted
    cp: /Users/sagarsodah/.esy/3_____________________________________________________________/i/opam__s__ocaml_secondary_compiler-opam__c__4.08.1-1-eca62758/share/ocaml-secondary-compiler/lib/threads/META: Operation not permitted
    mkdir: /Users/sagarsodah/.esy/3_____________________________________________________________/i/opam__s__ocaml_secondary_compiler-opam__c__4.08.1-1-eca62758/share/ocaml-secondary-compiler/lib/str: Operation not permitted
    cp: /Users/sagarsodah/.esy/3_____________________________________________________________/i/opam__s__ocaml_secondary_compiler-opam__c__4.08.1-1-eca62758/share/ocaml-secondary-compiler/lib/str: Operation not permitted
    make: *** [install-meta] Error 1
    error: command failed: 'make' 'install-meta' (exited with 2)
    esy-build-package: exiting with errors above...
    
  building @opam/ocamlfind-secondary@opam:1.9.1
esy: exiting due to errors above

aantron · 2021-09-25T12:07:11Z

@sagarjs This looks like an esy error (rather than Dream), and likely esy/esy#1062. Can you post the full esy build log to esy/esy#1062, including the part where it shows the dune and ocaml package versions built? cc @ManasJayanth

aantron · 2021-09-25T12:09:54Z

@sagarjs Please also include the esy version and the OS info, like at the top of the esy issue. Hopefully we can get this figured out quickly.

sagarjs · 2021-09-25T12:34:24Z

@aantron sure, i am using ocaml@4.12.0 though; should I still post the error in that issue?

ManasJayanth · 2021-09-25T14:28:25Z

@aantron

Yes, the resolution posted by @pm-mck is correct. And it just tested, esy-openssl builds fine on Windows, Macos M1 and Linux.
ocamlfind-secondary should never be depended on directly. Do you happen to recall why the examples suggest that users do so? Esy includes ocamlfind-secondary in the solution when the compiler is < 4.06 (There was a bug long ago where it was fetch for everyone, but has been fixed)

@sagarjs Can you try removing the ocamlfind-secondary and retry? With the following diff, the example built fine on my M1 mac

     "ocaml": "4.12.x"
   },
   "devDependencies": {
-    "@opam/ocaml-lsp-server": "*",
-    "@opam/ocamlfind-secondary": "*"
+    "@opam/ocaml-lsp-server": "*"
   },
   "resolutions": {
+    "esy-openssl": "esy-packages/esy-openssl#619ae2d46ca981ec26ab3287487ad98b157a01d1",
     "@opam/conf-libev": "esy-packages/libev:package.json#0b5eb6685b688649045aceac55dc559f6f21b829"
   },

sagarjs · 2021-09-25T14:59:43Z

@ManasJayanth thanks!
This worked. Even though I did not add esy-openssl in the resolutions section

ManasJayanth · 2021-09-25T18:38:01Z

Without the override, openssl 1g released is pulled in on my M1 mac (which doesn't have arm64 support). So, I'd say the override is important. I wonder what esy-openssl is actually getting resolved to over there, @sagarjs . Would you be able to share the esy.lock/index.json?

aantron · 2021-09-26T02:30:56Z

@ManasJayanth Thanks!

ocamlfind-secondary was added to the resolutions in #65 by @dangdennis, with the comment that it is necessary for OCaml Platform (the VSCode plugin) to do auto-completion. I don't know if that's accurate at this point — I am not currently using OCaml Platform, and haven't been able to set it up. @ManasJayanth, can you comment on the interaction between these resolutions and OCaml Platform?

dangdennis · 2021-09-26T03:01:56Z

I did not try without ocamlfind, but I’m glad to hear it’s not necessary. I’ll try it without now.

the error involving “make all”, I encountered too, and hinted at it in discord. I didn’t know how to debug. So I found a workaround by using an explicit previous minor version. But since we’re not supposed to use it directly, I’ll test it without.

My workaround was to pin ocamlfind/secondary to 1.8.1. Otherwise, we run into that weird make-all error. See my repo for a working example https://github.com/dangdennis/priority-box

Intel MacBook.

dangdennis · 2021-09-26T04:12:19Z

Removed ocamlfind and the vscode extension still works!

sagarjs · 2021-09-26T06:00:05Z

Without the override, openssl 1g released is pulled in on my M1 mac (which doesn't have arm64 support). So, I'd say the override is important. I wonder what esy-openssl is actually getting resolved to over there, @sagarjs . Would you be able to share the esy.lock/index.json?

@ManasJayanth
Pasting the esy-openssl part of the index.json below
I have installed opam using Rosetta2 Terminal because brew install opam did not work normally

    "esy-openssl@archive:https://github.com/openssl/openssl/archive/OpenSSL_1_1_1g.tar.gz#sha1:33324ff957edaae8ae575817b456320378da46ff@41b6fb3d": {
      "id":
        "esy-openssl@archive:https://github.com/openssl/openssl/archive/OpenSSL_1_1_1g.tar.gz#sha1:33324ff957edaae8ae575817b456320378da46ff@41b6fb3d",
      "name": "esy-openssl",
      "version":
        "archive:https://github.com/openssl/openssl/archive/OpenSSL_1_1_1g.tar.gz#sha1:33324ff957edaae8ae575817b456320378da46ff",
      "source": {
        "type": "install",
        "source": [
          "archive:https://github.com/openssl/openssl/archive/OpenSSL_1_1_1g.tar.gz#sha1:33324ff957edaae8ae575817b456320378da46ff"
        ]
      },
      "overrides": [ "esy.lock/overrides/41b6fb3db953254b0cb4c5e1b546f3f4" ],
      "dependencies": [
        "@opam/conf-pkg-config@opam:2@85f8644d",
        "@opam/conf-autoconf@github:esy-packages/esy-autoconf:package.json#fb93edf@d41d8cd9"
      ],
      "devDependencies": []
    },

ManasJayanth · 2021-09-26T06:37:02Z

I wonder if 1g works there because rosetta emulation - do you turn on intel emulation on while working on Dream (or in general)?

sagarjs · 2021-09-26T06:41:21Z

i have been running the examples without using the rosetta terminal
I have only run the examples till now

EDIT:
Sorry, i just noticed I made a really noob mistake (lols)
I have two terminals, one normal and one running rosetta compatiblity
I had named the terminals wrong and was using it the other way round since last few days :(

sagarjs · 2021-09-26T07:26:44Z

aantron · 2021-09-26T14:27:03Z

@sagarjs To confirm, you have it working with only adding the esy-openssl resolution, and you kept the ocamlfind-secondary dev dep, in your latest try?

pm-mck · 2021-09-27T16:14:21Z

I forgot to mention that I also had to remove ocamlfind-secondary.

I just performed the following on my M1:

Remove @opam/ocamlfind-secondary from 2-middleware/esy.json
Add "esy-openssl": "esy-packages/esy-openssl#619...." to resolutions in 2-middleware/esy.json
Run esy
Run esy start
Wait for step 4 to print and then curl http://localhost:8080
Curl printed "Good morning, world!"

I think this was a successful test. I'll cut a PR in a few moments.

aantron · 2021-09-28T10:08:13Z

Thanks to @pm-mck for applying the changes in the PR #163. And thanks to @pm-mck, @sagarjs, @ManasJayanth, and @dangdennis for figuring this issue out!

See #160.

sagarjs mentioned this issue Sep 25, 2021

Cannot use Dune 2 with OCaml < 4.07 esy/esy#1062

Closed

dangdennis mentioned this issue Sep 26, 2021

Add z-deploy-fly example #161

Merged

pm-mck mentioned this issue Sep 27, 2021

set the resolutions for Apple M1 build, remove ocamlfind-secondary for esy build in 2-middleware example #163

Merged

4 tasks

aantron closed this as completed in 34bd286 Sep 28, 2021

aantron added a commit that referenced this issue Sep 28, 2021

Propagate recent commit to example/w-esy/README.md

a1985d9

See #160.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Examples on Apple M1 need an additional resolution in esy. #160

Examples on Apple M1 need an additional resolution in esy. #160

pm-mck commented Sep 23, 2021

aantron commented Sep 24, 2021 •

edited

sagarjs commented Sep 25, 2021

aantron commented Sep 25, 2021

aantron commented Sep 25, 2021

sagarjs commented Sep 25, 2021

ManasJayanth commented Sep 25, 2021 •

edited

sagarjs commented Sep 25, 2021

ManasJayanth commented Sep 25, 2021 •

edited

aantron commented Sep 26, 2021

dangdennis commented Sep 26, 2021 •

edited

dangdennis commented Sep 26, 2021

sagarjs commented Sep 26, 2021

ManasJayanth commented Sep 26, 2021

sagarjs commented Sep 26, 2021 •

edited

sagarjs commented Sep 26, 2021

aantron commented Sep 26, 2021

pm-mck commented Sep 27, 2021

aantron commented Sep 28, 2021

Examples on Apple M1 need an additional resolution in esy. #160

Examples on Apple M1 need an additional resolution in esy. #160

Comments

pm-mck commented Sep 23, 2021

aantron commented Sep 24, 2021 • edited

sagarjs commented Sep 25, 2021

aantron commented Sep 25, 2021

aantron commented Sep 25, 2021

sagarjs commented Sep 25, 2021

ManasJayanth commented Sep 25, 2021 • edited

sagarjs commented Sep 25, 2021

ManasJayanth commented Sep 25, 2021 • edited

aantron commented Sep 26, 2021

dangdennis commented Sep 26, 2021 • edited

dangdennis commented Sep 26, 2021

sagarjs commented Sep 26, 2021

ManasJayanth commented Sep 26, 2021

sagarjs commented Sep 26, 2021 • edited

sagarjs commented Sep 26, 2021

aantron commented Sep 26, 2021

pm-mck commented Sep 27, 2021

aantron commented Sep 28, 2021

aantron commented Sep 24, 2021 •

edited

ManasJayanth commented Sep 25, 2021 •

edited

ManasJayanth commented Sep 25, 2021 •

edited

dangdennis commented Sep 26, 2021 •

edited

sagarjs commented Sep 26, 2021 •

edited