Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Renew iOS Fastlane match Profiles #8745

Closed
jakubgs opened this issue Aug 13, 2019 · 46 comments
Closed

Renew iOS Fastlane match Profiles #8745

jakubgs opened this issue Aug 13, 2019 · 46 comments

Comments

@jakubgs
Copy link
Member

jakubgs commented Aug 13, 2019

It appears our iOS Profiles created by Fastlane match are expiring today:
expired_ios_certs
This was spotted in #8722.

Your certificate 'XDR7F6R9W9.cer' is not valid, please check end date and renew it if necessary
@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

There is the --force flag that allows for renewal on every build:

Automatically renew your provisioning profiles to include all your devices using the --force option
https://docs.fastlane.tools/actions/match/#what-does-match-do-for-you
But we do not have that enabled, probably because we run multiple builds in parallel. Also sounds dodgy.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

According to closed and unresolved Fastlane issues:

Wtf, reading Fastlane match issues it seems like they are suggesting nuking profiles and re-creating them:

i guess the workaround would be to a nuke if it doesn't renew/recreate it
fastlane/fastlane#11663 (comment)

But then again:

I just solved it by manually deleting my debug certs and profiles for my app. I can't use nuke because I also have many certs and profiles for other apps on it. There's also the fact I don't like manually touching something that has been entirely generated by some system, as it is the case with the fastlane match repo.
fastlane/fastlane#10724 (comment)

1297882742677

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

I attempted to remove the certificate manually from the Apple Developer portal:
ios_removed_cert
But after re-running the PR iOS build for #8722 it failed with the same error:

Your certificate 'XDR7F6R9W9.cer' is not valid, please check end date and renew it if necessary

Which indicates to me that the https://github.com/status-im/ios-certificates repo has not been updated.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

I tried running:

fastlane match adhoc --force

On one of the macos CI slaves, but I'm just getting the same error.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

I could manually try removing the profiles/adhoc/AdHoc_im.status.ethereum.pr.mobileprovision from the ios-certificates repo and see if that makes fastlane match recognize the change and re-create the profile.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

I made this attempt in https://github.com/status-im/ios-certificates/commit/a822307fe08e5daa566e496b53abf9f8db39348e, lets see if this forces re-creation.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

That didn't work. Considering all our Profiles are managed by Fastlane match using fastlane nuke might be the best option.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

I've done the deed:

fastlane match nuke distribution

fastlane_match_nuke
And then re-created with:

fastlane match adhoc --force

Which seems to have worked, at least for one of the 3 profiles:
fastlane_match_adhoc
It appears the default length of Profile validity is 1 year, which isn't much, but I see no options to extend that in the documentation:
https://docs.fastlane.tools/actions/match/#parameters

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

And this seems to have created 2 commits in the ios-certificates repo:

Which seems to be what we want. I assume that during a new release build a new AppStore cert will be created as well.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

Well, now we're getting some new errors at least:

Signing Identity:     "iPhone Distribution: STATUS HOLDINGS PTE. LTD. (DTX7Z4U3YA)"
    /usr/bin/codesign --force --sign 676D290F0D4045A3310718D5AC1E22FC7D139ECB --preserve-metadata=identifier,entitlements,flags --timestamp=none /Users/jenkins/Library/Developer/Xcode/DerivedData/StatusIm-augjevvrecepxjcfmhxcoxextkui/Build/Intermediates.noindex/ArchiveIntermediates/StatusIm/InstallationBuildProductsLocation/Applications/StatusIm.app/Frameworks/QBImagePicker.framework
/Users/jenkins/Library/Developer/Xcode/DerivedData/StatusIm-augjevvrecepxjcfmhxcoxextkui/Build/Intermediates.noindex/ArchiveIntermediates/StatusIm/InstallationBuildProductsLocation/Applications/StatusIm.app/Frameworks/QBImagePicker.framework: errSecInternalComponent
Command /usr/bin/codesign failed with exit code 1

From: https://ci.status.im/job/status-react/job/prs/job/ios/job/PR-8722/32/consoleFull
Which is progress of a kind...

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

There was some other stuff there:

[15:15:54]: 🔓  Successfully decrypted certificates repo
[15:15:54]: Verifying that the certificate and profile are still valid on the Dev Portal...
Available session is not valid any more. Continuing with normal login.
[15:15:56]: Installing certificate...
[15:15:57]: There are no local code signing identities found.
You can run `security find-identity -v -p codesigning` to get this output.
This Stack Overflow thread has more information: https://stackoverflow.com/q/35390072/774.
(Check in Keychain Access for an expired WWDR certificate: https://stackoverflow.com/a/35409835/774 has more info.)
[15:15:57]: Setting key partition list... (this can take a minute if there are a lot of keys installed)
[15:15:58]: 
[15:15:58]: Could not configure imported keychain item (certificate) to prevent UI permission popup when code signing
Check if you supplied the correct `keychain_password` for keychain: `/Users/jenkins/Library/Keychains/login.keychain-db`
security: SecKeychainItemSetAccessWithPassword: The user name or passphrase you entered is not correct.
[15:15:58]: 
[15:15:58]: Please look at the following docs to see how to set a keychain password:
[15:15:58]:  - https://docs.fastlane.tools/actions/sync_code_signing
[15:15:58]:  - https://docs.fastlane.tools/actions/get_certificates

Of which the main part is:

Could not configure imported keychain item (certificate) to prevent UI permission popup when code signing
Check if you supplied the correct `keychain_password` for keychain: `/Users/jenkins/Library/Keychains/login.keychain-db`
security: SecKeychainItemSetAccessWithPassword: The user name or passphrase you entered is not correct.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

There's an issue that seems to show this error too:
fastlane/fastlane#14980

Apparently:

The problem is that the script is not matching the name of the key to the name of the certificate. In the output above the certificate is Cert_Project1_APN.cer and the key is Cert_Project2_APN.p12.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

Interestingly if I run the suggested security command on macos-02 I get 2 identities:

administrator@macos-02.ms-eu-dublin.ci.misc:~ % sudo su - jenkins
macos-02:~ jenkins$ security find-identity -v -p codesigning
  1) 123 "iPhone Distribution: STATUS HOLDINGS PTE. LTD. (DTX7Z4U3YA)"
  2) 789 "iPhone Distribution: STATUS HOLDINGS PTE. LTD. (DTX7Z4U3YA)"
     2 valid identities found

Which might be part of the problem. But that error shows up on macos-01 as well.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

But then again, there's also this:

Could not configure imported keychain item (certificate) to prevent UI permission popup when code signing
Check if you supplied the correct `keychain_password` for keychain: `/Users/jenkins/Library/Keychains/login.keychain-db`
security: SecKeychainItemSetAccessWithPassword: The user name or passphrase you entered is not correct.

Which would indicate that the keychain at /Users/jenkins/Library/Keychains/login.keychain-db was re-created with a new password, which could be true considering the creation time:

administrator@macos-01.ms-eu-dublin.ci.misc:~ % stat -x /Users/jenkins/Library/Keychains/login.keychain-db    
  File: "/Users/jenkins/Library/Keychains/login.keychain-db"
  Size: 7920864      FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (  502/ jenkins)  Gid: (  501/ jenkins)
Device: 1,4   Inode: 475373334    Links: 1
Access: Tue Aug 13 16:38:29 2019
Modify: Tue Aug 13 16:15:56 2019
Change: Tue Aug 13 16:15:56 2019

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

I've decided to re-create the login.keychain-db file:

 $ ansible ci-slave-macos --become-user jenkins -m shell -a 'rm -fr /Users/jenkins/Library/Keychains/* && security create-keychain -p "{{ jenkins_user_pass }}" /Users/jenkins/Library/Keychains/login.keychain-db'

macos-02.ms-eu-dublin.ci.misc | CHANGED | rc=0 >>
macos-03.ms-eu-dublin.ci.misc | CHANGED | rc=0 >>
macos-01.ms-eu-dublin.ci.misc | CHANGED | rc=0 >>

We'll see if that does anything.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

In this build the keychain access errors do not show up:
https://ci.status.im/blue/rest/organizations/jenkins/pipelines/status-react/pipelines/prs/pipelines/ios/branches/PR-8687/runs/29/nodes/107/steps/119/log/?start=0
But it still fails with errSecInternalComponent.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

It states:

Check if you supplied the correct `keychain_password` for keychain: `/Users/jenkins/Library/Keychains/login.keychain-db`

So I tried the password and it seems to work fine:

macos-01:~ jenkins$ security unlock-keychain -p $KEYCHAIN_PASSWORD /Users/jenkins/Library/Keychains/login.keychain-db
macos-01:~ jenkins$ security show-keychain-info /Users/jenkins/Library/Keychains/login.keychain-db
Keychain "/Users/jenkins/Library/Keychains/login.keychain-db" no-timeout
macos-01:~ jenkins$ security dump-keychain -a /Users/jenkins/Library/Keychains/login.keychain-db | grep alis
security: SecACLCopySimpleContents: The specified access control list is not in standard (simple) form.
security: SecACLCopySimpleContents: The specified access control list is not in standard (simple) form.
    "alis"<blob>="Apple Worldwide Developer Relations Certification Authority"
    "alis"<blob>="iPhone Distribution: STATUS HOLDINGS PTE. LTD. (DTX7Z4U3YA)"
    "alis"<blob>="Apple iPhone Certification Authority"
    "alis"<blob>="Apple iPhone OS Provisioning Profile Signing"
    "alis"<blob>="Apple Root Certificate Authority"

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

What a useless fucking error, what the FUCK does errSecInternalComponent mean?
Looking in /Users/jenkins/Library/Logs/gym/StatusIm-StatusIm.log gives me nothing as well.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

I opened the project in XCode to see if it was broken and I saw an error in the Signing section.
The error was about a non-existent Provisioning profile, so I selected an existing one:
xcode_signing
Resulting in changes to ios/StatusIm.xcodeproj/project.pbxproj that I've committed here 3f7a273 and tested in this manual iOS build:
https://ci.status.im/job/status-react/job/combined/job/mobile-ios/6685/
But that didn't work.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

I'm starting ton think that fastlane match generated a Profile with the wrong Certificate.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

Whenever I have to touch Apple code signing my quality of life and mental health drops down drastically.
SHOOT ME

@dmitryn dmitryn mentioned this issue Aug 13, 2019
@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

I tried adding verbose: true to match in Fastfile but that didn't show me anything useful:

15:43:07  DEBUG [2019-08-13 20:35:55.24]: Certificate '73PC2QL7CP.cer' is already installed on this machine
15:43:07  INFO [2019-08-13 20:35:55.24]: $ security import /var/folders/sn/6fj1hwmd11zdtjk4f5skyj2w0000gp/T/d20190813-27520-k2ubxy/certs/distribution/73PC2QL7CP.p12 -k '/Users/jenkins/Library/Keychains/login.keychain-db' -P '' -T /usr/bin/codesign -T /usr/bin/security
15:43:07  DEBUG [2019-08-13 20:35:55.29]: '73PC2QL7CP.p12' is already installed on this machine
...
15:43:07  INFO [2019-08-13 20:35:58.05]: Installing provisioning profile...
15:43:07  WARN [2019-08-13 20:35:58.93]: Setting environment variable 'sigh_im.status.ethereum_adhoc' to '7a9032a7-e304-4870-a781-078b01349c16'
15:43:07  WARN [2019-08-13 20:35:58.93]: Setting environment variable 'sigh_im.status.ethereum_adhoc_team-id' to 'DTX7Z4U3YA'
15:43:07  WARN [2019-08-13 20:35:58.93]: Setting environment variable 'sigh_im.status.ethereum_adhoc_profile-name' to 'match AdHoc im.status.ethereum'
15:43:07  WARN [2019-08-13 20:35:58.93]: Setting environment variable 'sigh_im.status.ethereum_adhoc_profile-path' to '/Users/jenkins/Library/MobileDevice/Provisioning Profiles/7a9032a7-e304-4870-a781-078b01349c16.mobileprovision'

It still fails with the same error:

Signing Identity:     "iPhone Distribution: STATUS HOLDINGS PTE. LTD. (DTX7Z4U3YA)"
    /usr/bin/codesign --force --sign 2CE8E560A721C4E4310D63D2E8D0168A2FE42FA8 --preserve-metadata=identifier,entitlements,flags --timestamp=none /Users/jenkins/Library/Developer/Xcode/DerivedData/StatusIm-gzbepwnrlqbukxbmmpfskimspfch/Build/Intermediates.noindex/ArchiveIntermediates/StatusIm/InstallationBuildProductsLocation/Applications/StatusIm.app/Frameworks/QBImagePicker.framework
/Users/jenkins/Library/Developer/Xcode/DerivedData/StatusIm-gzbepwnrlqbukxbmmpfskimspfch/Build/Intermediates.noindex/ArchiveIntermediates/StatusIm/InstallationBuildProductsLocation/Applications/StatusIm.app/Frameworks/QBImagePicker.framework: errSecInternalComponent

There's absolutely NOTHING useful in errSecInternalComponent. It can mean literally ANYTHING wrong with the signing setup on MacOS. Fuck my life.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

Apparently codesign command does have a --verbose flag:
https://www.manpagez.com/man/1/codesign/
But I have no idea how to make Fastlane use it. Even even with it I doubt I would get more info.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

Yuuuuuup, without --verbose:

/Users/jenkins/Library/Developer/Xcode/DerivedData/StatusIm-gzbepwnrlqbukxbmmpfskimspfch/Build/Intermediates.noindex/ArchiveIntermediates/StatusIm/InstallationBuildProductsLocation/Applications/StatusIm.app/Frameworks/QBImagePicker.framework:
errSecInternalComponent

With --verbose:

/Users/jenkins/Library/Developer/Xcode/DerivedData/StatusIm-gzbepwnrlqbukxbmmpfskimspfch/Build/Intermediates.noindex/ArchiveIntermediates/StatusIm/InstallationBuildProductsLocation/Applications/StatusIm.app/Frameworks/QBImagePicker.framework:
errSecInternalComponent

Just kill me.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

This is some great Apple documentation:

errSecInternalComponent
An internal component experienced an error.
Declaration
errSecInternalComponent = -2070
https://developer.apple.com/documentation/security/1542001-security_framework_result_codes/errsecinternalcomponent?language=objc

Oh... that explains everything...
how about a cup of rage

@jakubgs
Copy link
Member Author

jakubgs commented Aug 13, 2019

@jakubgs
Copy link
Member Author

jakubgs commented Aug 14, 2019

I found something that matches what I'm thinking:

The error comes from a mismatch in the provisioning profile setup and the certificates and the bundle id. Make sure that your PP, bundle id, and certificates are setup correctly in the and assigned correctly in itunes connect and in app.
http://www.itgo.me/a/x2795082655473005929/xcode-command-usr-bin-codesign-failed-with-exit-code-1-errsecinternalcomponen

So what I need to check is one by one all the relations between the Provisioning Profile, Bundle ID, and Certs.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 14, 2019

I also found this which suggests checking for expired ceritifcates:
http://ericholsinger.com/xcode-codesign-failed-with-exit-code-1-expired-certificates-causing-ambiguous-matches
But I checked that and found none expired.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 14, 2019

During the build Fastlane match prints these two sections:

+-------------------+-------------------------------------------------------------+
|                              Installed Certificate                              |
+-------------------+-------------------------------------------------------------+
| User ID           | DTX7Z4U3YA                                                  |
| Common Name       | iPhone Distribution: STATUS HOLDINGS PTE. LTD. (DTX7Z4U3YA) |
| Organisation Unit | DTX7Z4U3YA                                                  |
| Organisation      | STATUS HOLDINGS PTE. LTD.                                   |
| Country           | SG                                                          |
| Start Datetime    | 2019-08-13 17:53:57 UTC                                     |
| End Datetime      | 2020-08-12 17:53:57 UTC                                     |
+-------------------+-------------------------------------------------------------+
+---------------------+--------------------------------------------+----------------------------------------------------------------------------------------------------------------+
|                                                                          Installed Provisioning Profile                                                                           |
+---------------------+--------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| Parameter           | Environment Variable                       | Value                                                                                                          |
+---------------------+--------------------------------------------+----------------------------------------------------------------------------------------------------------------+
| App Identifier      |                                            | im.status.ethereum                                                                                             |
| Type                |                                            | adhoc                                                                                                          |
| Platform            |                                            | ios                                                                                                            |
| Profile UUID        | sigh_im.status.ethereum_adhoc              | 7a9032a7-e304-4870-a781-078b01349c16                                                                           |
| Profile Name        | sigh_im.status.ethereum_adhoc_profile-name | match AdHoc im.status.ethereum                                                                                 |
| Profile Path        | sigh_im.status.ethereum_adhoc_profile-path | /Users/jenkins/Library/MobileDevice/Provisioning Profiles/7a9032a7-e304-4870-a781-078b01349c16.mobileprovision |
| Development Team ID | sigh_im.status.ethereum_adhoc_team-id      | DTX7Z4U3YA                                                                                                     |
+---------------------+--------------------------------------------+----------------------------------------------------------------------------------------------------------------+

@jakubgs
Copy link
Member Author

jakubgs commented Aug 14, 2019

@jakubgs
Copy link
Member Author

jakubgs commented Aug 14, 2019

Same goes for the info on the Apple Developer portal:

@jakubgs
Copy link
Member Author

jakubgs commented Aug 14, 2019

I just noticed something! The Certificate type on the site is iOS Distribution, but the value for CODE_SIGN_IDENTITY is iPhone Distribution. I wonder if this makes a difference.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 14, 2019

I attempted to fix this using this commit: 4222014
And ran this build: https://ci.status.im/job/status-react/job/combined/job/mobile-ios/6688
But I got back:

Code Signing Error: No certificate for team 'DTX7Z4U3YA' matching 'iOS Distribution' found:  Select a different signing certificate for CODE_SIGN_IDENTITY, a team that matches your selected certificate, or switch to automatic provisioning.

So that doesn't seem to be the issue.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 14, 2019

Based on what I found in this issue, I attempted to allow all apps on our CI hosts to access those certs.
fastlane/fastlane#10589

Steps:

  1. Open Keychain Access app under Utilities in Finder:
    keychain_01
  2. Find the certificate you want Fastlane to use:
    keychain_02
  3. Double-click Imported private key and go to the Access Control tab:
    keychain_03
  4. Select the Allow all applications to access this item option:
    keychain_04
  5. Click Save Changes and confirm by inputting the user password:
    keychain_05

Let's see if that does it.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 14, 2019

That didn't do shit... but I have another idea based on this comment:
fastlane/fastlane#10589 (comment)
Based on which I've implemented this:

# helper for proper teradown of resource
def with(ctx)
    yield ctx.setup
ensure
    ctx.teardown
end

# This is a helper for match and signing to avoid having to
# approve of the keychain access through the UI dialog.
# Details: https://github.com/status-im/status-react/issues/8745
class Keychain
    attr_accessor :name, :pass
    @@pass = "temppassword"

    def initialize(name)
        # We use epoch time to void clashes with CI builds
        @name = "#{name}_#{Time.now.to_f}.keychain-db"
        @path = "~/Library/Keychains/#{@name}"
        Fastlane::Actions::CreateKeychainAction.run(
            name: @name,
            password: @@pass,
            timeout: false,
        )
    end

    # for use in with()
    def setup
        self
    end

    def teardown
        if not File.exist? File.expand_path(@path)
            raise "Keychain file missing: #{@path}"
        end
        Fastlane::Actions::DeleteKeychainAction.run(
            name: @name
        )
    end
end

So now we can run match and build_ios_app inside this with() helper:

with Keychain.new('adhoc') do |kc|
    match()
    build_ios_app()
end

I'm trying this in the use-temp-keychain branch.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 15, 2019

Okay, it appears the approach with the temporary keychain works, but only partially. Despite Fastlane match using it, as we can see here:

+-----------------------+-----------------------------------------------+
|                       Summary for match 2.128.0                       |
+-----------------------+-----------------------------------------------+
| type                  | adhoc                                         |
| verbose               | true                                          |
| force_for_new_devices | true                                          |
| readonly              | false                                         |
| keychain_name         | adhoc_1565827254.833005.keychain-db           |

It still attempts to import the WWDR(Apple Worldwide Developer Relations Certificate Authority) certificate into login.keychain-db:

Installing certificate...
$ security list-keychains -d user
▸     "/Users/jenkins/Library/Keychains/login.keychain-db"
$ security find-certificate -c 'Apple Worldwide Developer Relations Certification Authority' /Users/jenkins/Library/Keychains/login.keychain-db
▸ keychain: "/Users/jenkins/Library/Keychains/login.keychain-db"

Which I think is what causes the need for the UI interaction prompt for the password.

So when I removed the /Users/jenkins/Library/Keychains/login.keychain-db file from the CI host the build failed with new error:

security: SecKeychainSearchCopyNext: The specified item could not be found in the keychain.
$ security list-keychains -d user
$ security default-keychain -d user
security: SecKeychainCopyDomainDefault user: A default keychain could not be found.
Installing WWDR Cert:
  curl -f -o /var/folders/pc/sgwfk7115_7b8dndw3xgz_0r0000gp/T/AppleWWDRCA20190815-14358-1uc6mqg https://developer.apple.com/certificationauthority/AppleWWDRCA.cer && \
  security import /var/folders/pc/sgwfk7115_7b8dndw3xgz_0r0000gp/T/AppleWWDRCA20190815-14358-1uc6mqg 
Failed to install WWDR Certificate, checking output to see why

Which might mean that Fastlane requires login.keychain-db to exist. Unless I can change its path.

Links:

@jakubgs
Copy link
Member Author

jakubgs commented Aug 15, 2019

This error comes from here:
https://github.com/fastlane/fastlane/blob/dd889fc2cbaae8f91ae9a099e71b9077dcae84dd/fastlane_core/lib/fastlane_core/cert_checker.rb#L85
The location of the keychain where WWDR is imported is determined by this method:
https://github.com/fastlane/fastlane/blob/dd889fc2cbaae8f91ae9a099e71b9077dcae84dd/fastlane_core/lib/fastlane_core/cert_checker.rb#L92-L105

    def self.wwdr_keychain
      priority = [
        "security list-keychains -d user",
        "security default-keychain -d user"
      ]
      priority.each do |command|
        keychains = Helper.backticks(command, print: FastlaneCore::Globals.verbose?).split("\n")
        unless keychains.empty?
          # Select first keychain name from returned keychains list
          return keychains[0].strip.tr('"', '')
        end
      end
      return ""
    end

Which for Jenkins clearly is login.keychain-db:

administrator@macos-01.ms-eu-dublin.ci.misc:~ % sudo su - jenkins
security default-keychain -d usermacos-01:~ jenkins$ security default-keychain -d user
    "/Users/jenkins/Library/Keychains/login.keychain-db"

Unless it doesn't exist:

macos-01:~ jenkins$ rm /Users/jenkins/Library/Keychains/login.keychain-db
macos-01:~ jenkins$ security default-keychain -d user
security: SecKeychainCopyDomainDefault user: A default keychain could not be found.

So it seems the only way would be to temporarily set the default user keychain to the one we created for the sake of the build with Keychain class using:

macos-01:~ jenkins$ security default-keychain --help
default-keychain: illegal option -- -
Usage: default-keychain [-d user|system|common|dynamic] [-s [keychain]]
    -d  Use the specified preference domain
    -s  Set the default keychain to the specified keychain

Which sounds kinda dodgy considering we run multiple builds on the same host.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 15, 2019

Actually, this is even simpler because create_keychain has a default_keychain flag:

default_keychain
Should the newly created Keychain be the new system default keychain
https://docs.fastlane.tools/actions/create_keychain/

@jakubgs
Copy link
Member Author

jakubgs commented Aug 15, 2019

It does work:

administrator@macos-02.ms-eu-dublin.ci.misc:~ % sudo su - jenkins
macos-02:~ jenkins$ security default-keychain -d user
    "/Users/jenkins/Library/Keychains/adhoc_1565830300.859616.keychain-db"

But the question is how will this affect parallel builds on the same host? This might be a form of a weird race condition.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 15, 2019

Looks like even despite that change I still get the same errSecInternalComponent error:
https://ci.status.im/job/status-react/job/combined/job/mobile-ios/6717/console

@jakubgs
Copy link
Member Author

jakubgs commented Aug 15, 2019

And despite using a temporary and unlocked from the start keychain, I still get the password prompt from codesign if I run the failing build step via a VNC session:

codesign_password_prompt

I really hoped I could finally fix this prompt issue by using a temporary keychain, but apparently not.
I will open an issue with Fastlane about this tomorrow, or now I'm just going to manually fix the prompt issue on macos-02 by putting in the password and clicking Always Allow.

EDIT: What's even weirder is that after I click Always Allow it prompts for the password AGAIN!

@jakubgs
Copy link
Member Author

jakubgs commented Aug 15, 2019

I've applied the manual fix to all macos-* hosts.

@jakubgs
Copy link
Member Author

jakubgs commented Aug 15, 2019

Okay, so here is what I learned from this... exercise in productive and well documented madness:

  • Fastlane match calls codesign which requires the Keychain file to be unlocked via UI
  • This UI unlock can be achieved by clicking Always Allow when prompted for access
  • Unlocking process is certificate specific and a new certificate requires Always Allow trick again
  • The codesign utility does not seem to care if the Keychain file has been unlocked in advance
    • True for both for using security unlock-keychan and creating an unlocked keychain with Fastlane
  • Even when using a separate Keychain fastlane still tries to access the default one to import WWDR

I'm not sure if this weird behavior is due to how Fastlane uses codesign, or codesign just being all kinds of fucked up. I will open a detailed issue with Fastlane tomorrow.

I love working with Apple products... I need some sleep.
1345153875415

@jakubgs
Copy link
Member Author

jakubgs commented Aug 15, 2019

Opened an issue with Fastlane: fastlane/fastlane#15185
Closing.

@rromanchuk
Copy link

@jakubgs i dont know how i got here, since i have no match/signing issues, or why i read this entire thread, but damn, what a masterpiece.

Not only did i enjoy it, but this is kind of indexable troubleshooting is what i imagine must be like to debug software in heaven. Web searchable, insightful, entertaining, and complete.

Now i want a collection of your debugging in this format, technology agnostic without any organization or additional context in coffee table book print format. I would non ironically read it cover to cover.

@jakubgs
Copy link
Member Author

jakubgs commented Jul 22, 2021

@rromanchuk thanks, appreciate it. The thing is, when dealing with issues like that you HAVE to create detailed notes, otherwise you'll either accidentally stumble on a solution - hightly unlikely - or you'll just go insane through an endless process of trial and error without any learning occurring.

If you like this, you might like some of my other hits:

Most of my research is unfortunately hidden in various private repos, since I work with infrastructure, and that most of the time has to be kept private, even if the repos themselves do not contain any actual secrets.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants