Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reunification with NixOS/nixpkgs #312

Merged
merged 338 commits into from Oct 9, 2018
Merged

Reunification with NixOS/nixpkgs #312

merged 338 commits into from Oct 9, 2018

Conversation

matthewbauer
Copy link
Member

@matthewbauer matthewbauer commented Jun 8, 2018

Work in progress reunification branch.

Uses Nixpkgs commit: NixOS/nixpkgs@f3fc04b

Current Status

Currently GHC 8.4 works great. You can build desktop apps for macOS & Linux works as well as Linux->Android and macOS->iOS. We have been able to build a few different apps built on obelisk this way.

On the other hand, GHCJS 8.4 is still not ready yet. There have been some changes internally in GHC that GHCJS has not kept up with, leading to some critical issues. We can build a GHCJS app such as reflex-todomvc, but only when the text-jsstring is disabled. When text-jsstring is enabled, two issues occur:

The text-jsstring is important for us to get working & inlined correctly to ensure the generated JavaScript is as performant as possible.

@ElvishJerricco
Copy link
Contributor

Excellent! :) Super glad to see movement on this front.

I think it would be a good idea to merge this only once all the changes needed to nixpkgs are upstreamed. See #313

@matthewbauer
Copy link
Member Author

matthewbauer commented Jun 9, 2018

I am just going to post this here in case someone know what this comes from:

setupCompilerEnvironmentPhase
Build with /nix/store/s0dg1c9ivx4cp33f0h1inm8br2hj36ay-ghcjs-8.2.0.1.
unpacking sources
unpacking source archive /nix/store/gdp22vlxb63nnvzljada3smqn1rmigzb-distributive-0.5.3.tar.gz
source root is distributive-0.5.3
setting SOURCE_DATE_EPOCH to timestamp 1501288672 of file distributive-0.5.3/Warning.hs
patching sources
Replace Cabal file with edited version from http://hackage.haskell.org/package/distributive-0.5.3/revision/5.cabal.
Run jailbreak-cabal to lift version restrictions on build inputs.
compileBuildDriverPhase
setupCompileFlags: -package-db=/tmp/nix-build-distributive-0.5.3.drv-2/package.conf.d -j8 -threaded
GHC_PACKAGE_PATH=/tmp/nix-build-distributive-0.5.3.drv-2/setup-package.conf.d:
[1 of 2] Compiling Warning          ( Warning.hs, /tmp/nix-build-distributive-0.5.3.drv-2/Warning.o )
[2 of 2] Compiling Main             ( Setup.lhs, /tmp/nix-build-distributive-0.5.3.drv-2/Main.o )

Setup.lhs:24:1: warning: [-Wdeprecations]
    Module ‘Warning’:
      You are configuring this package without cabal-doctest installed.
      The doctests test-suite will not work as a result.
      To fix this, install cabal-doctest before configuring.
   |
24 | import Warning ()
   | ^^^^^^^^^^^^^^^^^
Linking Setup ...
configuring
configureFlags: --verbose --prefix=/nix/store/03c3sykrznpc6v6ifb8jx04rzrcm2f9x-distributive-0.5.3 --libdir=$prefix/lib/$compiler --libsubdir=$pkgid --docdir=/nix/store/s0n98ggyhl4gx3796w30pkkwd71phllj-distributive-0.5.3-doc/share/doc --with-gcc=gcc --package-db=/tmp/nix-build-distributive-0.5.3.drv-2/package.conf.d --ghc-option=-j8 --disable-split-objs --enable-library-profiling --profiling-detail=all-functions --disable-profiling --enable-shared --disable-coverage --disable-executable-dynamic --disable-tests --enable-library-vanilla --enable-library-for-ghci --ghc-option=-split-sections --ghcjs --ghcjs-options=-fexpose-all-unfoldings --extra-lib-dirs=/nix/store/z1ila4azfjpjq4vwhpbwq97q7jay1yll-cabal-doctest-1.0.6/lib --extra-lib-dirs=/nix/store/w1zndp7wc5glygwqrn7abzggg9cg8c49-ncurses-6.1/lib --extra-lib-dirs=/nix/store/18fj72dj4nc1lx0pfidkxyrzm91hx5qh-gmp-6.1.2/lib
Configuring distributive-0.5.3...
Flags chosen: tagged=True, semigroups=True
Dependency base -any: using base-4.10.1.0
Dependency base-orphans -any: using base-orphans-0.7
Dependency tagged >=0.7 && <1: using tagged-0.8.5
Dependency transformers -any: using transformers-0.5.2.0
Dependency transformers-compat -any: using transformers-compat-0.5.1.4
Source component graph: component lib
Configured component graph:
    component distributive-0.5.3-9wERQg7K8jyA6SgZYWDfBN
        include base-4.10.1.0
        include base-orphans-0.7-2XyGKJbZ0tFRlibTJmrXi
        include transformers-0.5.2.0
        include transformers-compat-0.5.1.4-EFxO4sa1gaR4ZJSTM5ZOeh
        include tagged-0.8.5-DFDUhteBUV67fGgw064nCS
Linked component graph:
    unit distributive-0.5.3-9wERQg7K8jyA6SgZYWDfBN
        include base-4.10.1.0
        include base-orphans-0.7-2XyGKJbZ0tFRlibTJmrXi
        include transformers-0.5.2.0
        include transformers-compat-0.5.1.4-EFxO4sa1gaR4ZJSTM5ZOeh
        include tagged-0.8.5-DFDUhteBUV67fGgw064nCS
        Data.Distributive=distributive-0.5.3-9wERQg7K8jyA6SgZYWDfBN:Data.Distributive,Data.Distributive.Generic=distributive-0.5.3-9wERQg7K8jyA6SgZYWDfBN:Data.Distributive.Generic
Ready component graph:
    definite distributive-0.5.3-9wERQg7K8jyA6SgZYWDfBN
        depends base-4.10.1.0
        depends base-orphans-0.7-2XyGKJbZ0tFRlibTJmrXi
        depends transformers-0.5.2.0
        depends transformers-compat-0.5.1.4-EFxO4sa1gaR4ZJSTM5ZOeh
        depends tagged-0.8.5-DFDUhteBUV67fGgw064nCS
Warning:
    This package indirectly depends on multiple versions of the same package. This is very likely to cause a compile failure.
      package transformers (transformers-0.5.2.0) requires base-4.10.1.0
      package distributive (distributive-0.5.3) requires base-4.10.1.0
      package transformers-compat (transformers-compat-0.5.1.4-EFxO4sa1gaR4ZJSTM5ZOeh) requires base-4.10.1.0-DuxFDcnl7HKKK3wxhYafB1
      package transformers (transformers-0.5.2.0-HtEf5nV5vOXJgl7pipqNkj) requires base-4.10.1.0-DuxFDcnl7HKKK3wxhYafB1
      package template-haskell (template-haskell-2.12.0.0-Eyc5VNYLRr62krNFdMFA2P) requires base-4.10.1.0-DuxFDcnl7HKKK3wxhYafB1
      package tagged (tagged-0.8.5-DFDUhteBUV67fGgw064nCS) requires base-4.10.1.0-DuxFDcnl7HKKK3wxhYafB1
      package pretty (pretty-1.1.3.3-C0ohXD3PAbYEH1bCEDflEy) requires base-4.10.1.0-DuxFDcnl7HKKK3wxhYafB1
      package ghc-boot-th (ghc-boot-th-8.2.2-DTRLqtrKgRNDP2PY9r0n5) requires base-4.10.1.0-DuxFDcnl7HKKK3wxhYafB1
      package deepseq (deepseq-1.4.3.0-CGHPJ73R5W952UnsBn6B8R) requires base-4.10.1.0-DuxFDcnl7HKKK3wxhYafB1
      package base-orphans (base-orphans-0.7-2XyGKJbZ0tFRlibTJmrXi) requires base-4.10.1.0-DuxFDcnl7HKKK3wxhYafB1
      package array (array-0.5.2.0-IVj6PzCng6rEeSmnkJVc4c) requires base-4.10.1.0-DuxFDcnl7HKKK3wxhYafB1
      package integer-gmp (integer-gmp-1.0.1.0) requires ghc-prim-0.5.1.1
      package base (base-4.10.1.0) requires ghc-prim-0.5.1.1
      package transformers-compat (transformers-compat-0.5.1.4-EFxO4sa1gaR4ZJSTM5ZOeh) requires ghc-prim-0.5.1.1-7UiAmegvHVuFzwVDFOv8fE
      package pretty (pretty-1.1.3.3-C0ohXD3PAbYEH1bCEDflEy) requires ghc-prim-0.5.1.1-7UiAmegvHVuFzwVDFOv8fE
      package integer-gmp (integer-gmp-1.0.1.0-ESU36R2qYi71eTQfsks4mM) requires ghc-prim-0.5.1.1-7UiAmegvHVuFzwVDFOv8fE
      package base-orphans (base-orphans-0.7-2XyGKJbZ0tFRlibTJmrXi) requires ghc-prim-0.5.1.1-7UiAmegvHVuFzwVDFOv8fE
      package base (base-4.10.1.0-DuxFDcnl7HKKK3wxhYafB1) requires ghc-prim-0.5.1.1-7UiAmegvHVuFzwVDFOv8fE
      package base (base-4.10.1.0) requires integer-gmp-1.0.1.0
      package base (base-4.10.1.0-DuxFDcnl7HKKK3wxhYafB1) requires integer-gmp-1.0.1.0-ESU36R2qYi71eTQfsks4mM
      package distributive (distributive-0.5.3) requires transformers-0.5.2.0
      package transformers-compat (transformers-compat-0.5.1.4-EFxO4sa1gaR4ZJSTM5ZOeh) requires transformers-0.5.2.0-HtEf5nV5vOXJgl7pipqNkj
      package tagged (tagged-0.8.5-DFDUhteBUV67fGgw064nCS) requires transformers-0.5.2.0-HtEf5nV5vOXJgl7pipqNkj
Using Cabal-2.0.1.0 compiled by ghc-8.2
Using compiler: ghcjs-8.2.0.1
Using install prefix:
/nix/store/03c3sykrznpc6v6ifb8jx04rzrcm2f9x-distributive-0.5.3
Executables installed in:
/nix/store/03c3sykrznpc6v6ifb8jx04rzrcm2f9x-distributive-0.5.3/bin
Libraries installed in:
/nix/store/03c3sykrznpc6v6ifb8jx04rzrcm2f9x-distributive-0.5.3/lib/ghcjs-8.2.0.1/distributive-0.5.3
Dynamic Libraries installed in:
/nix/store/03c3sykrznpc6v6ifb8jx04rzrcm2f9x-distributive-0.5.3/lib/ghcjs-8.2.0.1/x86_64-linux-ghcjs-8.2.0.1-ghc8_2_2
Private executables installed in:
/nix/store/03c3sykrznpc6v6ifb8jx04rzrcm2f9x-distributive-0.5.3/libexec/x86_64-linux-ghcjs-8.2.0.1-ghc8_2_2/distributive-0.5.3
Data files installed in:
/nix/store/03c3sykrznpc6v6ifb8jx04rzrcm2f9x-distributive-0.5.3/share/x86_64-linux-ghcjs-8.2.0.1-ghc8_2_2/distributive-0.5.3
Documentation installed in:
/nix/store/s0n98ggyhl4gx3796w30pkkwd71phllj-distributive-0.5.3-doc/share/doc
Configuration files installed in:
/nix/store/03c3sykrznpc6v6ifb8jx04rzrcm2f9x-distributive-0.5.3/etc
No alex found
Using ar found on system at:
/nix/store/frjbmg85g8y2k25klck6djnnmc7k8ipj-binutils-2.30/bin/ar
No c2hs found
No cpphs found
No doctest found
Using gcc version 7.3.0 given by user at:
/nix/store/ik57r8n32sf5xrvn7m0yb1cvfzm5bmj8-gcc-wrapper-7.3.0/bin/gcc
Using ghc version 8.2.2 found on system at:
/nix/store/b5ldrzwcz4jzwvsygsinxk7m8nf787pk-ghc-8.2.2/bin/ghc
Using ghc-pkg version 8.2.2 found on system at:
/nix/store/b5ldrzwcz4jzwvsygsinxk7m8nf787pk-ghc-8.2.2/bin/ghc-pkg
Using ghcjs version 8.2.0.1 found on system at:
/nix/store/s0dg1c9ivx4cp33f0h1inm8br2hj36ay-ghcjs-8.2.0.1/bin/ghcjs
Using ghcjs-pkg version 8.2.2 found on system at:
/nix/store/s0dg1c9ivx4cp33f0h1inm8br2hj36ay-ghcjs-8.2.0.1/bin/ghcjs-pkg
No greencard found
Using haddock version 2.18.1 found on system at:
/nix/store/s0dg1c9ivx4cp33f0h1inm8br2hj36ay-ghcjs-8.2.0.1/bin/haddock-ghcjs
No happy found
Using haskell-suite found on system at: haskell-suite-dummy-location
Using haskell-suite-pkg found on system at: haskell-suite-pkg-dummy-location
No hmake found
Using hpc version 0.67 found on system at:
/nix/store/b5ldrzwcz4jzwvsygsinxk7m8nf787pk-ghc-8.2.2/bin/hpc
Using hsc2hs version 0.68.2 found on system at:
/nix/store/s0dg1c9ivx4cp33f0h1inm8br2hj36ay-ghcjs-8.2.0.1/bin/hsc2hs-ghcjs
Using hscolour version 1.24 found on system at:
/nix/store/fw0icg7q4hybs6j0ci1kxf1sv428aaca-hscolour-1.24.4/bin/HsColour
No jhc found
Using ld found on system at:
/nix/store/ik57r8n32sf5xrvn7m0yb1cvfzm5bmj8-gcc-wrapper-7.3.0/bin/ld
No lhc found
No lhc-pkg found
No pkg-config found
Using runghc version 8.2.2 found on system at:
/nix/store/b5ldrzwcz4jzwvsygsinxk7m8nf787pk-ghc-8.2.2/bin/runghc
Using strip version 2.30 found on system at:
/nix/store/frjbmg85g8y2k25klck6djnnmc7k8ipj-binutils-2.30/bin/strip
Using tar found on system at:
/nix/store/avmg2kv205l2wd4zdj547r5dp3vbkj6i-gnutar-1.30/bin/tar
No uhc found
*** abort because of serious configure-time warning from Cabal
builder for '/nix/store/ijx53yic0v5s2akacbc79lsjcix0mmrc-distributive-0.5.3.drv' failed with exit code 1

All of the version look correct - but not Cabal still give that error.

@templateK
Copy link

@matthewbauer
#312 (comment)
ekmett's libraries uses custom setup which depends on doctest package. I suspected that this somehow causes multiple dependency error. I was able to compile those packages by removing the custom setup.

#312 (comment)
In the path /nix/store/...apple-framework-CoreText,CoreText and Resources are symbolic link but Headers which contains CTLine.h is not. this leads clang into incorrect path searching.
I think the bug is on the nixos apple-framework. I could compile jsaddle-wkwebview by changing the include path to absolute one.

@matthewbauer matthewbauer changed the title [wip] Reunification [wip] Reunification with NixOS/nixpkgs Jun 10, 2018
@matthewbauer
Copy link
Member Author

ekmett's libraries uses custom setup which depends on doctest package. I suspected that this somehow causes multiple dependency error. I was able to compile those packages by removing the custom setup.

I don't think it's related to doctest - tests for distributive should be disabled already in reflex-platform. I really am not sure what's going on though. I've opened an issue in Nixpkgs (where it also shows up) here: NixOS/nixpkgs#41806.

@matthewbauer
Copy link
Member Author

matthewbauer commented Jun 13, 2018

@matthewbauer
Copy link
Member Author

I also think the iOS logic needs to be reworked but will leave that out for now. Latest Nixpkgs should not have to look for the /Applications/Xcode.app directory because it now uses requireFile.

default.nix Outdated
};
};
};
let config = { allowUnfree = true; };
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use allowUnfreePredicate so that we only allow the unfree derivations that we actually need.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this was originally set, I'm not going to try to modify it here for consistency. But definitely would be worth looking into using allowUnfreePredicate instead of allowUnfree.

@matthewbauer
Copy link
Member Author

matthewbauer commented Jun 14, 2018

So most things are now working! Currently I've gotten reflex-todomvc built on:

  • Web (ghcjs.reflex-todomvc)
  • Android (androidReflexTodomvc)
  • macOS (ghc.reflex-todomvc)
  • Linux (ghc.reflex-todomvc)

We are still waiting for iOS and there may be some changes necessary here.

  • iOS (iosReflexTodomvc)

iOS fails on GHC with:

make[1]: *** No rule to make target 'libffi/build/inst/lib/libffi.dylib', needed by 'rts/dist/build/libffi.dylib'.  Stop.
make[1]: *** Waiting for unfinished jobs....
"inplace/bin/ghc-pkg" --simple-output field rts library-dirs \
  | sed -e 's/\([^ ][^ ]*\)/-L\1/g' >> rts/dist/libs.depend
make: *** [Makefile:127: all] Error 2
builder for '/nix/store/jz40qfgprd72s0zpm9as7j53km82sij2-aarch64-apple-ios-ghc-8.4.2.drv' failed with exit code 2

@matthewbauer matthewbauer force-pushed the reunification branch 3 times, most recently from 4e96fc6 to d847258 Compare June 15, 2018 15:30
@alexfmpe
Copy link
Member

Note that iOS simulator is broken in both sides of the reunification, so it shouldn't block this

@matthewbauer matthewbauer force-pushed the reunification branch 2 times, most recently from 9611c2b to 2561c83 Compare June 15, 2018 17:45
@matthewbauer matthewbauer changed the title [wip] Reunification with NixOS/nixpkgs Reunification with NixOS/nixpkgs Jun 15, 2018
@matthewbauer
Copy link
Member Author

matthewbauer commented Jun 15, 2018

Ok everything seems to be working now! iOS is building but I have not had the chance to test the app & need to set up the team id & certificate stuff. The new rebased reunification Nixpkgs is at:

https://github.com/obsidiansystems/nixpkgs/tree/reunification3

We now have very few differences with Nixpkgs master:

Staging has merged into master so NixOS/nixpkgs@645f03b is now available!

Also ghcWithPackages is still broken & the reflexEnv stuff will not work. See NixOS/nixpkgs#42032. @Ericson2314 thinks he will have a fix for this soon. There is some issue with how the package-db is generated in Nixpkgs.

The jsaddle version in this tree points to this currently open PR: ghcjs/jsaddle#67. I hope to get it merged soon.

I really need to get the Hydra job running again but there is some evaluation error. Does anyone know how to fix it? This is the evaluation error: https://hydra.reflex-frp.org/jobset/reflex-platform/reflex-platform-pr-312#tabs-errors

@matthewbauer matthewbauer force-pushed the reunification branch 9 times, most recently from 8ad6514 to b649401 Compare June 15, 2018 23:18
Ericson2314 and others added 22 commits September 28, 2018 18:18
Do this to make sure we have all reflex profiler fixes
Thankfully, most of these changes are cherry-picks from commits that
made it into 8.4. The little at the end that isn't is a) stuff that
should be upstreamed for all versions, and b) a quick hack just
disabling some linker routines and iserv.
That way they can be installed at the same time
See comment. We couldn't start due to a missing symbol.
Enable the Reflex profiler in all profiled builds
for ios mobile fixes
@luigy luigy merged commit 1771661 into develop Oct 9, 2018
@luigy luigy deleted the reunification branch October 9, 2018 23:02
@Ericson2314
Copy link
Member

Finally!

@ryantrinkle
Copy link
Member

ryantrinkle commented Oct 10, 2018 via email

@nh2
Copy link

nh2 commented Mar 25, 2020

Hey,

this commit d9eed03 - dontCheck ghcjs-base - disables tests.

This seems to have masked that ghcjs upstream tests have been failing for a long time (years?) because nobody seems to actually run them.

In particular, we see this one failing:

https://github.com/ghcjs/ghcjs-base/blob/6be0e992e292db84ab42691cfb172ab7cd0e709e/test/Tests/Marshal.hs#L88

pureMarshalTestGroup (TypeName "[Char]"   :: TypeName [Char]  ),
  [Char]:
    pure_to_from_jsval: [Failed]
*** Failed! Falsified (after 6 tests and 2 shrinks):
"\750560"
"\29664" /= "\750560"
(used seed -1085708459)
    pure_to_from_jsval_maybe: [Failed]
*** Failed! Falsified (after 5 tests and 2 shrinks):
Just "\1043892"
Just "\60852" /= Just "\1043892"
(used seed 61512362)
    to_from_jsval: [Failed]
*** Failed! Falsified (after 4 tests):
"\877237"
"\25269" /= "\877237"
(used seed -1440847798)
    to_from_jsval_maybe: [Failed]
*** Failed! Falsified (after 4 tests and 2 shrinks):
Just "\994408"
Just "\11368" /= Just "\994408"
(used seed 1579843250)
    to_from_jsval_list: [Failed]
*** Failed! Falsified (after 3 tests and 2 shrinks):
["\1005757"]
["\22717"] /= ["\1005757"]
(used seed 2053171004)
    to_from_jsval_list_maybe: [Failed]
*** Failed! Falsified (after 7 tests and 6 shrinks):
[Just "\529263"]
[Just "\4975"] /= [Just "\529263"]
(used seed 729348745)
    to_from_jsval_list_list: [OK, passed 1 tests]
    to_from_jsval_maybe_list: [Failed]
*** Failed! Falsified (after 4 tests and 1 shrink):
Just ["\689068"]
Just ["\33708"] /= Just ["\689068"]

That seems scary, because rountripping strings between Haskell and Javascript is somewhat important.

Any idea what this could be?

@nh2
Copy link

nh2 commented Mar 25, 2020

@luigy Is it possible that this has anything to do with your recent string-related changes in https://github.com/ghcjs/ghcjs-base/commits/master ?

@Ericson2314
Copy link
Member

@nh2 I think in general there is a bunch of ghcjs stuff here that needs to get upstreamed into nixpkgs. We might have indeed disabled too many tests, I suspect WIP versions of the "text-jsstring" patch had something to do with it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

10 participants