Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support MacPorts (installing and running) #429

Open
tgc-dk opened this issue Sep 20, 2018 · 10 comments
Open

Support MacPorts (installing and running) #429

tgc-dk opened this issue Sep 20, 2018 · 10 comments
Labels
Application Compatibility Darling needs fixes for compatibility with a specific Application

Comments

@tgc-dk
Copy link

tgc-dk commented Sep 20, 2018

I've build and installed the latest darling master on an up to date Archlinux.

Installing MacPorts

Being optimistic I tried to install the MacPorts pkg, but the darling installer failed due to a missing "title" attribute in some "choice" tags in the bundled XML. I was able to workaround this by using xar to extract the pkg, edit the XML (copied the id attribute content to a new title attribute) and create a new pkg. This new pkg was installable using the darling installer, though some warnings/errors is displayed:

$ installer –pkg MacPorts-2.5.3-10.12-Sierra-title-fixed.pkg –target /
 
installer: Package name is MacPorts
installer: Installing selected choice default
installer: Installing selected choice org.macports.MacPorts
The MacPorts Project, postflight script version 2.4: checking the shell environment for user "tgc".
Creating group "macports"
./postinstall: line 160: /usr/sbin/dseditgroup: No such file or directory
Creating user "macports"
./postinstall: line 165: /usr/bin/dscl: No such file or directory
./postinstall: line 168: /usr/bin/dscl: No such file or directory
./postinstall: line 174: /usr/bin/dscl: No such file or directory
./postinstall: line 175: /usr/bin/dscl: No such file or directory
./postinstall: line 176: /usr/bin/dscl: No such file or directory
./postinstall: line 177: /usr/bin/dscl: No such file or directory
./postinstall: line 179: /usr/bin/dscl: No such file or directory
./postinstall: line 180: /usr/bin/dscl: No such file or directory
./postinstall: line 181: /usr/bin/dscl: No such file or directory
./postinstall: line 181: /usr/bin/dscl: No such file or directory
./postinstall: line 182: /usr/bin/dscl: No such file or directory
./postinstall: line 183: /usr/bin/dscl: No such file or directory
./postinstall: line 193: /usr/bin/dscl: No such file or directory
Copying archive_sites.conf.default to archive_sites.conf
Copying macports.conf.default to macports.conf
Copying pubkeys.conf.default to pubkeys.conf
Copying sources.conf.default to sources.conf
Copying variants.conf.default to variants.conf
 
Removing old man pages...
Done.
 
./postinstall: line 213: /usr/bin/dscl: No such file or directory
An attempt to determine your shell name failed! Please set your MacPorts compatible environment manually.
Adding [default] tag to sources.conf if needed...
Updating port image format...
could not read attributes of "/opt/local/var/macports": operation not supported
    while executing
"file attributes $portdbpath -hidden"
    (procedure "mportinit" line 525)
    invoked from within
"mportinit ui_options"
    (file "./images_to_archives.tcl" line 15)
could not read attributes of "/opt/local/var/macports": operation not supported
    while executing
"file attributes $portdbpath -hidden"
    (procedure "mportinit" line 525)
    invoked from within
"mportinit ui_options"
    (file "./dedup_portfiles.tcl" line 14)
could not read attributes of "/opt/local/var/macports": operation not supported
    while executing
"file attributes $portdbpath -hidden"
    (procedure "mportinit" line 525)
    invoked from within
"mportinit"
    (file "./cxx_stdlib_overridden.tcl" line 11)
Synchronizing the MacPorts installation with the project's rsync server...
could not read attributes of "/opt/local/var/macports": operation not supported
    while executing
"file attributes $portdbpath -hidden"
    (procedure "mportinit" line 525)
    invoked from within
"mportinit ui_options global_options global_variations"
Error: /opt/local/bin/port: Failed to initialize MacPorts, could not read attributes of "/opt/local/var/macports": operation not supported
An attempt to synchronize your recent MacPorts installation with the project's rsync server failed!
Please run 'sudo port -d selfupdate' manually to find out the cause of the error.
installer: Installing package org.macports.MacPorts version 0.2.5.3.0.0.0.0.0 (13613 KB)
installer: Extracting files
The script has terminated with exit code 1

Running MacPorts

Since macport did actually install I then (of course) tried to run it:

$ /opt/local/bin/port
could not read attributes of "/opt/local/var/macports": operation not supported
    while executing
"file attributes $portdbpath -hidden"
    (procedure "mportinit" line 525)
    invoked from within
"mportinit ui_options global_options global_variations"
Error: /opt/local/bin/port: Failed to initialize MacPorts, could not read attributes of "/opt/local/var/macports": operation not supported

I will admit that I did not install Xcode as normally required for MacPorts since I hoped to use MacPorts binary packages, without compiling. So some of the errors shown above might disappear if Xcode is installed, though I think the installer should support the macport pkg format (without title attributes in choice tags).

@bugaevc
Copy link
Member

bugaevc commented Sep 20, 2018

Hi, yes, it'd be great to get MacPorts to work.

The choice-title thing is tracked at #366. The state of user/groups conformance is indeed quite dismal (and package managers are the ones that care most about this), see #354, we'd appreciate any ideas how to make this work better. (Having some version of dscl would be a first step.)

As for the mportinit error, it seems we don't support the "hidden" attribute on files, that one should be easy to fix...

@ahyattdev ahyattdev added the Application Compatibility Darling needs fixes for compatibility with a specific Application label Nov 25, 2018
@TheBrokenRail
Copy link
Contributor

Installing now gives this error:

Darling [~/Downloads]$ installer -pkg MacPorts-2.6.2-10.15-Catalina.pkg -target /
installer: Package name is MacPorts
installer: Installing selected choice default
installer: Installing selected choice org.macports.MacPorts
installer: Installing package org.macports.MacPorts version 0.2.6.2.0.0.0.0.0 (13864 KB)
installer: Extracting files
The MacPorts Project, postflight script version 2.6: checking the shell environment for user "connor".
Creating group "macports"
./postinstall: line 160: /usr/sbin/dseditgroup: No such file or directory
Creating user "macports"
./postinstall: line 165: /usr/bin/dscl: No such file or directory
./postinstall: line 168: /usr/bin/dscl: No such file or directory
./postinstall: line 174: /usr/bin/dscl: No such file or directory
./postinstall: line 175: /usr/bin/dscl: No such file or directory
./postinstall: line 176: /usr/bin/dscl: No such file or directory
./postinstall: line 177: /usr/bin/dscl: No such file or directory
./postinstall: line 179: /usr/bin/dscl: No such file or directory
./postinstall: line 180: /usr/bin/dscl: No such file or directory
./postinstall: line 181: /usr/bin/dscl: No such file or directory
./postinstall: line 181: /usr/bin/dscl: No such file or directory
./postinstall: line 182: /usr/bin/dscl: No such file or directory
./postinstall: line 183: /usr/bin/dscl: No such file or directory
./postinstall: line 193: /usr/bin/dscl: No such file or directory
Copying archive_sites.conf.default to archive_sites.conf
Copying macports.conf.default to macports.conf
Copying pubkeys.conf.default to pubkeys.conf
Copying sources.conf.default to sources.conf
Copying variants.conf.default to variants.conf

Removing old man pages...
Done.

./postinstall: line 213: /usr/bin/dscl: No such file or directory
An attempt to determine your shell name failed! Please set your MacPorts compatible environment manually.
Adding [default] tag to sources.conf if needed...
dyld: lazy symbol binding failed: Symbol not found: ____chkstk_darwin
  Referenced from: /opt/local/libexec/macports/lib/libtcl8.5.dylib
  Expected in: /usr/lib/libSystem.B.dylib

dyld: Symbol not found: ____chkstk_darwin
  Referenced from: /opt/local/libexec/macports/lib/libtcl8.5.dylib
  Expected in: /usr/lib/libSystem.B.dylib

Abort trap: 6 (core dumped)

and running gives this error:

Darling [~/Downloads]$ /opt/local/bin/port
dyld: lazy symbol binding failed: Symbol not found: ____chkstk_darwin
  Referenced from: /opt/local/libexec/macports/lib/libtcl8.5.dylib
  Expected in: /usr/lib/libSystem.B.dylib

dyld: Symbol not found: ____chkstk_darwin
  Referenced from: /opt/local/libexec/macports/lib/libtcl8.5.dylib
  Expected in: /usr/lib/libSystem.B.dylib

Abort trap: 6 (core dumped)

@TheBrokenRail
Copy link
Contributor

This might be related to https://reviews.llvm.org/D40856.

@LubosD
Copy link
Member

LubosD commented Feb 27, 2020

Yes, we need to implement sys_getattrlist(at). MacPorts and Tcl suck for not having a fallback for ENOTSUP returned from this API.

This is an optional API the underlying filesystem may not support. Pretty much all of Apple's code falls back to (l)stat(at) upon getting ENOTSUP.

LubosD added a commit that referenced this issue Feb 27, 2020
@LubosD
Copy link
Member

LubosD commented Feb 27, 2020

This is the current state. We need to implement more of CoreCrypto:

$ sudo /opt/local/bin/port install mc
Warning: configured user/group macports does not exist, will build as root
struct ccrng_state *ccDRBGGetRngState(int *) just putting this here until we build CommonCrypto
struct ccrng_state *ccDRBGGetRngState(int *) just putting this here until we build CommonCrypto
Segmentation fault: 11

I don't have a backtrace yet, but this is the crash:

[359745.278025] tclsh8.5[3251056]: segfault at 18 ip 00007ffff70240dd sp 00007fffffdf8c38 error 4 in libobjc.A.dylib[7ffff6ff5000+3dd000]

@LubosD
Copy link
Member

LubosD commented Feb 28, 2020

Current state after libsystem_trace fix:

    frame #7: 0x00007ffff798cece libsystem_c.dylib`abort at abort.c:91
    frame #8: 0x00007ffff7ab6ed1 libsystem_malloc.dylib`szone_error(szone=0x00007ffff6679000, is_corruption=1, msg="incorrect checksum for freed object - object was probably modified after being freed.", ptr=0x00007fbc85c1a518, fmt=0x0000000000000000) at magazine_malloc.c:116
    frame #9: 0x00007ffff7ad15c0 libsystem_malloc.dylib`free_list_checksum_botch(szone=0x00007ffff6679000, ptr=0x00007fbc85c1a518) at magazine_inline.h:129
    frame #10: 0x00007ffff7acd06f libsystem_malloc.dylib`tiny_malloc_from_free_list [inlined] free_list_unchecksum_ptr(szone=0x00007ffff6679000, ptr=0x00007fbc85c1a518) at magazine_inline.h:190
    frame #11: 0x00007ffff7acceb7 libsystem_malloc.dylib`tiny_malloc_from_free_list(szone=0x00007ffff6679000, tiny_mag_ptr=0x00007ffff6663a00, mag_index=0, msize=12) at magazine_tiny.c:1630
    frame #12: 0x00007ffff7acde82 libsystem_malloc.dylib`tiny_malloc_should_clear(szone=0x00007ffff6679000, msize=12, cleared_requested=0) at magazine_tiny.c:1845
    frame #13: 0x00007ffff7ab7e2c libsystem_malloc.dylib`szone_malloc_should_clear(szone=0x00007ffff6679000, size=184, cleared_requested=0) at magazine_malloc.c:477
    frame #14: 0x00007ffff7ab833f libsystem_malloc.dylib`szone_malloc(szone=0x00007ffff6679000, size=184) at magazine_malloc.c:513
    frame #15: 0x00007ffff7ad7ab8 libsystem_malloc.dylib`default_zone_malloc(zone=0x00007ffff6679000, size=184) at malloc.c:217
    frame #16: 0x00007ffff7ad482a libsystem_malloc.dylib`malloc_zone_malloc(zone=0x00007ffff7aeb000, size=184) at malloc.c:1343
    frame #17: 0x00007ffff7b7e414 CoreFoundation`__CFAllocatorSystemAllocate(size=184, hint=0, info=0x00007ffff7aeb000) at CFBase.c:239
    frame #18: 0x00007ffff7b7d6f5 CoreFoundation`CFAllocatorAllocate(allocator=0x00007ffff7d7adb0, size=184, hint=0) at CFBase.c:620
    frame #19: 0x00007ffff7b80721 CoreFoundation`__CFBasicHashRehash [inlined] __CFBasicHashAllocateMemory(ht=0x00007fbc85c1fe10, count=23, elem_size=8, strong='\0', compactable='\0') at CFBasicHash.c:284
    frame #20: 0x00007ffff7b80604 CoreFoundation`__CFBasicHashRehash(ht=0x00007fbc85c1fe10, newItemCount=1) at CFBasicHash.c:1153
    frame #21: 0x00007ffff7b84b45 CoreFoundation`__CFBasicHashAddValue(ht=0x00007fbc85c1fe10, bkt_idx=11, stack_key=140737215189944, stack_value=140737351497704) at CFBasicHash.c:1262
    frame #22: 0x00007ffff7b81fc7 CoreFoundation`CFBasicHashAddValue(ht=0x00007fbc85c1fe10, stack_key=140737215189944, stack_value=140737351497704) at CFBasicHash.c:1346
    frame #23: 0x00007ffff7be37d1 CoreFoundation`CFDictionaryAddValue(hc=0x00007fbc85c1fe10, key=0x00007fffefb7d3b8, value=0x00007ffff7d7b7e8) at CFDictionary.c:595
    frame #24: 0x00007fffef4d187d Security`SecPolicyAddBasicX509Options(options=0x00007fbc85c1fe10) at SecPolicy.c:1728
    frame #25: 0x00007fffef4c9ed1 Security`SecPolicyCreateBasicX509 at SecPolicy.c:1874
    frame #26: 0x00007fffef4d8635 Security`SecTrustCreateWithCertificates(certificates=0x00007fbc85c28b30, policies=0x0000000000000000, trust=0x00007fffffdf95e8) at SecTrust.c:212
    frame #27: 0x00007fffef45261c Security`tls_helper_set_peer_pubkey(hdsk=0x00007fbc56261400) at tls_helpers.c:197
    frame #28: 0x00007fffef973f85 Security`tls_handshake_message_callback(ctx=0x00007fbc85c0e5b0, event=tls_handshake_message_certificate) at tlsCallbacks.c:96
    frame #29: 0x00007ffff7b18dda libsystem_coretls.dylib`SSLProcessHandshakeRecordInner(rec=(length = 2897, data = "\v"), ctx=0x00007fbc56261400) at sslHandshake.c:241
    frame #30: 0x00007ffff7b1966d libsystem_coretls.dylib`SSLProcessHandshakeRecord(rec=(length = 2897, data = "\v"), ctx=0x00007fbc56261400) at sslHandshake.c:491
    frame #31: 0x00007ffff7b233f4 libsystem_coretls.dylib`tls_handshake_process(filter=0x00007fbc56261400, message=(length = 2897, data = "\v"), contentType='\x16') at tls_handshake.c:298
    frame #32: 0x00007fffef9752d5 Security`SSLHandshakeProceed(ctx=0x00007fbc85c0e5b0) at sslTransport.c:508
    frame #33: 0x00007fffef975bfb Security`SSLHandshake(ctx=0x00007fbc85c0e5b0) at sslTransport.c:430
    frame #34: 0x00007ffff4274220 libcurl.4.dylib`darwinssl_connect_step2(conn=0x00007fbc565c7600, sockindex=0) at darwinssl.c:1828
    frame #35: 0x00007ffff42722e8 libcurl.4.dylib`darwinssl_connect_common(conn=0x00007fbc565c7600, sockindex=0, nonblocking=true, done=0x00007fffffdfa39d) at darwinssl.c:2156
    frame #36: 0x00007ffff4272008 libcurl.4.dylib`Curl_darwinssl_connect_nonblocking(conn=0x00007fbc565c7600, sockindex=0, done=0x00007fffffdfa39d) at darwinssl.c:2192
    frame #37: 0x00007ffff4270eab libcurl.4.dylib`Curl_ssl_connect_nonblocking(conn=0x00007fbc565c7600, sockindex=0, done=0x00007fffffdfa39d) at vtls.c:322
    frame #38: 0x00007ffff422dfa4 libcurl.4.dylib`https_connecting(conn=0x00007fbc565c7600, done=0x00007fffffdfa39d) at http.c:1403
    frame #39: 0x00007ffff4269da8 libcurl.4.dylib`Curl_protocol_connecting(conn=0x00007fbc565c7600, done=0x00007fffffdfa39d) at url.c:3537
    frame #40: 0x00007ffff423fff6 libcurl.4.dylib`multi_runsingle(multi=0x00007fbc85c0b410, now=(tv_sec = 1582900874, tv_usec = 674307), data=0x00007fbc565ada00) at multi.c:1251
    frame #41: 0x00007ffff423f5a2 libcurl.4.dylib`curl_multi_perform(multi_handle=0x00007fbc85c0b410, running_handles=0x00007fffffdfa4a8) at multi.c:1788
    frame #42: 0x00007ffff42172a4 libcurl.4.dylib`easy_transfer(multi=0x00007fbc85c0b410) at easy.c:715
    frame #43: 0x00007ffff4216704 libcurl.4.dylib`easy_perform(data=0x00007fbc565ada00, events=false) at easy.c:803
    frame #44: 0x00007ffff42165a7 libcurl.4.dylib`curl_easy_perform(easy=0x00007fbc565ada00) at easy.c:822
    frame #45: 0x00007ffff42bb3b7 Pextlib.dylib`CurlGetSizeCmd + 945
    frame #46: 0x00007ffff42bba1d Pextlib.dylib`CurlCmd + 220

@LubosD
Copy link
Member

LubosD commented Feb 28, 2020

So, repeatedly using libcurl definitely uncovers this memory corruption. Something within corecrypto is probably stepping out of bounds. This is something for @ahyattdev.

Either way, without working corecrypto, MacPorts support won't move forward.

#include <curl/curl.h>

int main()
{
        while(1)
        {
                CURL *curl = curl_easy_init();
                if(curl)
                {
                        CURLcode res;
                        curl_easy_setopt(curl, CURLOPT_URL, "https://packages.macports.org");
                        res = curl_easy_perform(curl);
                        curl_easy_cleanup(curl);
                }
        }
}

@bugaevc
Copy link
Member

bugaevc commented Feb 28, 2020

So, repeatedly using libcurl definitely uncovers this memory corruption. Something within corecrypto is probably stepping out of bounds. This is something for @ahyattdev.

This is simple (with libgmalloc 🙂). ccrsa_verify_pkcs1v15() forgets to call ccn_sizeof_n() on the size before passing it to malloc():

https://github.com/darlinghq/darling-corecrypto/blob/d055f73ea001044db286e56d9d1ffdd76a8ea7b7/src/ccrsa.c#L107-L109

Compare to this just above:

https://github.com/darlinghq/darling-corecrypto/blob/d055f73ea001044db286e56d9d1ffdd76a8ea7b7/src/ccrsa.c#L84-L89

On another note, this function leaks the buffers on errors. Would be better off allocating them on the stack — I believe CoreCrypto is not supposed to rely on malloc (it should be usable on bare metal without a heap).

@bugaevc
Copy link
Member

bugaevc commented Feb 28, 2020

Proposed patch to fix the above:

--- a/src/ccrsa.c
+++ b/src/ccrsa.c
@@ -85,7 +85,7 @@ int ccrsa_verify_pkcs1v15(ccrsa_pub_ctx_t key, const uint8_t *oid,
        cc_size sig_units = ccn_nof_size(sig_len);
        cc_size sig_bytes = ccn_sizeof_n(sig_units);
 
-       cc_unit *s = malloc(sig_bytes);
+       cc_unit *s = alloca(sig_bytes);
        if (ccn_read_uint(sig_units, s, sig_len, sig))
        {
 #if DEBUG
@@ -104,14 +104,15 @@ int ccrsa_verify_pkcs1v15(ccrsa_pub_ctx_t key, const uint8_t *oid,
                return 1;
        }
 
-       cc_size m_size = ccn_sizeof_size(digest_len);
-       cc_unit *m = malloc(m_size);
-       ccn_read_uint(m_size, m, digest_len, digest);
+       cc_size digest_units = ccn_sizeof_size(digest_len);
+       cc_size digest_bytes = ccn_sizeof_n(digest_units);
+       cc_unit *m = alloca(digest_bytes);
+       ccn_read_uint(digest_units, m, digest_len, digest);
 
        // m = s^e mod n
-       
 
-       struct cczp *zp = malloc(cczp_size(m_size));
+
+       struct cczp *zp = alloca(cczp_size(digest_units));
        CCZP_N(zp) = ccn_nof_size(digest_len);
        // maybe we should copy modulus into cczp_prime
        cc_unit *zp_mod = (void *)zp + sizeof(struct cczp) /*+ ccn_sizeof_n(1)*/;
@@ -120,17 +121,11 @@ int ccrsa_verify_pkcs1v15(ccrsa_pub_ctx_t key, const uint8_t *oid,
        ccn = zp_mod;
        cczp_init(zp);
        //void cczp_power(cczp_const_t zp, cc_unit *r, const cc_unit *m, const cc_unit *e);
-       
-       cc_unit *r = malloc(sig_bytes);
+
+       cc_unit *r = alloca(sig_bytes);
        cczp_power(zp, r, m, exponent);
 
        printf("we get here \n");
-
-       free(s);
-       free(zp);
-       free(m);
-
-
 }
 
 /*

then we crash in cczp_power() because ws (the pointer) this is obviously uninitialized:

https://github.com/darlinghq/darling-corecrypto/blob/f9c046900cdaa9ab52c68535d3f413410562cb5b/src/cczp.c#L113-L116

(malloc shouldn't be used there either).

@bugaevc
Copy link
Member

bugaevc commented Feb 28, 2020

--- a/src/cczp.c
+++ b/src/cczp.c
@@ -20,7 +20,7 @@ void cczp_mod(cczp_const_t zp, cc_unit *r, const cc_unit *s2n, cc_ws_t ws)
        /*
 if D = 0 then error(DivisionByZeroException) end
 Q := 0                  -- Initialize quotient and remainder to zero
-R := 0                     
+R := 0
 for i := n <E2><88><92> 1 .. 0 do  -- Where n is number of bits in N
   R := R << 1           -- Left-shift R by 1 bit
   R(0) := N(i)          -- Set the least-significant bit of R equal to bit i of the numerator
@@ -133,21 +133,19 @@ void cczp_power(cczp_const_t zp, cc_unit *r, const cc_unit *m,
        }
 
        // TODO: Assert :: (modulus - 1) * (modulus - 1) does not overflow base
-       
+
        // Result is still one at this point
-       
-       cc_unit *base = malloc(n_sizeof*2);
+
+       cc_unit *base = alloca(n_sizeof*2);
        memset(base, 0, n_sizeof*2);
        memcpy(base, m, n_sizeof);
 
        // Use a workspace size 2n+1 because ws.end is non-inclusive end pointer
-       cc_ws_t ws;
-       ws->start = malloc(n*2+1);
-       ws->end = ws->start+(n*2);
+       cc_ws ws;
+       ws.start = alloca(n*2+1);
+       ws.end = ws.start+(n*2);
 
        // ccn integers are little endian, so it's OK that the 3rd parameter
        // is length 2n (despite not needing it)
-       cczp_mod_prime(zp)(zp, base, base, ws);
-
-       free(ws->start);
+       cczp_mod_prime(zp)(zp, base, base, &ws);
 }

With this, the example seems to no longer crash, whether run with gmalloc or not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Application Compatibility Darling needs fixes for compatibility with a specific Application
Projects
None yet
Development

No branches or pull requests

5 participants