[New Feature / Proposal] Add new required attribute in podspec to provide privacy details

[mfaani]

This PR description has been edited to take into account the feedback that was later given by comments below. You can find the original posting here

• I've read and understood the CONTRIBUTING guidelines and have done my best effort to follow.

• Authors: @prohoney, @AliSoftware
• Date: 1 Jan, 2021

Motivation

Starting from December 8, 2020, Apple has required new apps and app updates to include App privacy details on the App Store. App owners have to go through every pod and all its dependency's readmes or if nothing is revealed then reach out to pod creators to just figure out what information the pod collects. The current process is not transparent nor easy.

Proposal

Each Pod should mention their privacy details, so upon pod install / pod update a report file is generated where the different categories along with the purposes of every pod used are mentioned for each target. To acheive this the following changes need to be made:

PodSpec changes

Add a new array attribute to the the podspec named privacy_details

Expected structure for this attribute (Inspiration: [1])

s.privacy_details = [
  {
    "category": "NAME",
    "purposes": [
      "PRODUCT_PERSONALIZATION",
      "APP_FUNCTIONALITY"
    ],
    "data_protections": [
      "DATA_LINKED_TO_YOU"
    ]
  },
  {
    "category": "PURCHASE_HISTORY",
    "purposes": [
      "APP_FUNCTIONALITY"
    ],
    "data_protections": [
      "DATA_LINKED_TO_YOU",
      "DATA_USED_TO_TRACK_YOU"
    ]
  }
]

💡 Tip: Given that PodSpecs are ruby DSLs, pod owners can easily choose to provide the content of that attribute by reading it from a file (especially given that it might be more verbose than other typical PodSpec attributes).

We might want to mention that tip in the attribute documentation (and blog post that will accompany this new feature) especially since not everybody realizes that PodSpecs are interpreted ruby and can contain ruby code like this to make the spec more readable in some cases.

s.privacy_details = JSON.parse(File.read("privacy.json"))

Handling Subspecs

Note that the new privacy_details attribute will be allowed both at the root level of the podspec as well as for the definition of each subspec, if any. In such cases, the resolved privacy details for each subspec will be the result of merging the value of attribute defined at the root level (if one is defined there) with the value defined at the subspecies level.

Pod::Spec.new do |s|
  s.name = 'CoolKit'
  s.privacy_details = JSON.parse(File.read("common-privacy-details.json")) # includes NAME, EMAIL
  
  s.subspec 'SoundKit' do |cs|
    cs.dependency 'Alamofire'
    cs.privacy_details = JSON.parse(File.read("soundkit-privacy-details.json")) # includes AUDIO
  end

  s.subspec 'Model' do |ms|
    ms.privacy_details = JSON.parse(File.read("model-privacy-details.json")) # includes SENSITIVE_INFO 
  end
end

Then the privacy detail you get upon installation of a pod depends on which subspecies gets included:

• The whole pod, e.g. pod 'CoolKit' (by default it would install all pods): NAME, EMAIL, AUDIO, SENSITIVE_INFO.
• Only pod 'CoolKit/SoundKit': NAME, EMAIL, AUDIO.
• Only pod 'CoolKit/Model': NAME, EMAIL, SENSITIVE_INFO.

If your intention is that if only SoundKit is installed then only AUDIO to be reported as your privacy detail, then you have to restructure your PodSpec accordingly, and remove s.privacy_details from the spec's root and instead only add NAME and EMAIL onto the Model's sub specs.

Example of a Podspec when not all specs share common privacy details

Pod::Spec.new do |s|
  s.name = 'CoolKit'
  
  s.subspec 'SoundKit' do |cs|
    cs.dependency 'Alamofire'
    cs.privacy_details = JSON.parse(File.read("soundkit-privacy-details.json")) # includes AUDIO
  end

  s.subspec 'Model' do |ms|
    ms.privacy_details = JSON.parse(File.read("model-privacy-details.json")) # includes NAME, EMAIL, SENSITIVE_INFO 
  end
end

Gradual rollout for requiring the attribute

• New pods or any update should include the new privacy_details attribute in each of their subspecs. Mention which of the 32 data types (along with its purpose and data protection) they are collecting. If privacy_details isn't stated, then a message will be written to stdout .
• It’s gonna be a warning for now but we plan to make it an error in a future version of CocoaPods.
  • The warning would be like this:

  WARN | privacy_details: Pods (X,Y,Z) did not specify privacy_details. This will be a requirment in the upcomming releasees.

Versioning Pods

• We should make a recommendation to the community that if there is a change in the privacy_details then patches are not welcomed. At least a minor version update should happen.

Generated Report file

• Upon pod install or pod update, a json is generated for each target of the Podfile.
• The file will be located where other Support files are at: $PROJECT_DIRECTORY/Pods/Target Support Files/Pods-$target-name/privacy_details.json
• Generation of the report file (a.k.a privacay.json / privacyDetails.json) is defaulted to on.
• In order to increase awarness of the generation of this report file a messege is written to stdout upon pod install

Generated privacy details into
INFO | /pods/Target Support Files/pods-targetA/privacyDetails.json
INFO | /pods/Target Support Files/pods-targetB/privacyDetails.json

• If you don’t want it then you have to opt out with a new flag e.g. pod install --no-privacy-details.

Example of a multi-target report file

/// $PROJECT_DIRECTORY/Pods/Target Support Files/Pods-targetFoo/privacy_details.json

   {
    "privacy_details": [
      {
        "category": "NAME",
        "purposes": [
          "PRODUCT_PERSONALIZATION",
          "APP_FUNCTIONALITY"
        ],
        "data_protections": [
          "DATA_LINKED_TO_YOU"
        ],
        "collectors": ["pod1", "pod2"]
      },
      {
        "category": "PURCHASE_HISTORY",
        "purposes": [
          "APP_FUNCTIONALITY"
        ],
        "data_protections": [
          "DATA_LINKED_TO_YOU",
          "DATA_USED_TO_TRACK_YOU"
        ],
        "collectors": ["pod1", "pod5"]
      },
      {
        "data_protections": [
          "DATA_NOT_COLLECTED"
        ],
        "collectors": ["pod3", "pod6"] 
      }
    ],
    "unknown_privacy_details": ["pod4", "pod7"],
  }

/// $PROJECT_DIRECTORY/Pods/Target Support Files/Pods-targetBar/privacy_details.json

  "target2": {
    "privacy_details": [
      {
        "category": "NAME",
        "purposes": [
          "PRODUCT_PERSONALIZATION",
          "APP_FUNCTIONALITY"
        ],
        "data_protections": [
          "DATA_LINKED_TO_YOU"
        ],
        "collectors": ["pod1", "pod2", "pod8"]
      }
    ]
  }

Note: Every included permutation (of data types, purpose and data protection) should contain a key named "collectors". This provides app owners granularity to see what data each pod is collecting.

Podfile changes

• In regards to the app's own privacy details we can:
  • Do nothing. Dev can figure out the rest in regards to how to add privacy details about what data their own app is collecting
  • Add new attribute into Podfile. Name it target_privacy_details. App owners can then add their privacy details. Then upon pod install/update do a union with what’s generated from the pods.
    • The target_privacy_details can be set either a root option or a target configuration. The value provided for the target configuration will merge with the root option value if set.
• Make a recommendation for app owners that if there is a change in the privacy_details then at least a minor version update should happen (not just a patch bump).

Backwards Compatibility

• Older versions published to CocoaPods would continue to work without their privacy details
• New pods or updates to existing pods would require a value for their privacy details
• The ultimate PrivacyDetails.json would mention all the values for updated pods and include a key named unknown_privacy_details (included in the sample report file) for the (old) pods that don't have it set.

Ways to bypass this

• Given that CocoaPods is open source, pod owners can alter the validations that pod trunk push performs
• Yet it's a step forward, towards transparency and privacy. It cautions app owners to think again when a pod doesn't include privacy details.
• At then end a pod owner may have malicious intent or simply forget to mention something they log or keep track of. This should lead app owners to create issues with a privacyDetails label on repos. It will take some time from the community to get use to this process.

Other notes

• We're looking for feedbacks on this and see if maybe the CocoaPods team have been working on this. If not we're keen to implement the PR for this feature and would like to know what files to look into to start development on this

• Based on my mediocre Ruby knowledge, is the following a good starting point:
  add a new attribute to Pod::Specification::DSL like this:

attribute :privacy_details,
  :container   => Array,
  :singularize => true,
  :inherited => true

and then fix any error that I get through pod ipc spec <podspec>

---

[1]: On Dec 3rd, 2020, fastlane added a new action upload_app_data_usage_to_app_store (PR link). It requires a json with a similar shape, which can include a single category or multiple or none. The format/structure of the JSON we're using in this issue/proposal is directly inspired by the one expected by fastlane, both for consistency and so that it provides us the additional benefit of interoperability, as users could then use the JSON generated by CocoaPods via this new feature to feed it as an input for their fastlane actions directly 👌

[freak4pc]

Smart idea in my opinion. Would allow aggregating all of these details from third parties using fastlane, as you mentioned, to make sure your privacy settings are update.

An even nicer idea could be warning you that installing this pod will change your App's privacy details.

[orta]

If this was a first party file format that Apple + Xcode supported, I think I'd be onboard with the level of ecosystem changes this request has. However, as you have to use Fastlane to get any of the advantages of this scale of automation, I'm not sure it's worth breaking the deployment of every current podspec.

The idea is good in general, perhaps it could maybe be a trunk warning?

[AliSoftware]

@orta you don't have to use Fastlane at all to get the benefit of this feature.

If you don't use Fastlane then you will still benefit from the JSON generated by CocoaPods at the end of pod install because you'll be able to read that generated JSON to know what to answer when you fill the privacy details form manually on the dev portal. Even if you don't automate that privacy details submission with fastlane but instead submit that form manually you'll still benefit from the feature and from the report than CocoaPods would generate from the privacy details of each pod.

We could totally decide on a completely different file format for the report (the JSON that pod install will generate listing the merged privacy details for all integrated pods) and also a completely different format for the podspec attribute. I'm fine with using a different format if you think something else would fit better. We happened to choose that format because (1) we had to choose something anyway (2) we didn't want to reinvent the wheel and have to think of all the possibilities and what we needed to be able to represent while another bunch of smart people (from the fastlane OSS project) already thought about the problem before us (3) and the fact that we end up incidentally use the same format than fastlane uses happens to give an extra bonus for people who have to use fastlane anyway, without removing any benefit from people who don't.

If we decide on another format for the podspec attribute and/or the generated report (eg generate the report as a YAML to match the fact that other files generated by CocoaPods, like the lockfile, are also YAML) it would work just the same and provide the exact same benefit, ie allow the users to end up with a nice human readable report they could use to know what to answer when they fill the privacy report form in ADC. The only difference is that people who happen to also use Fastlane (witch is a lot of iOS devs in practice) would need to write an additional script to transform that YAML (or whatever we choose as a report format) into the JSON that Fastlane uses for its action.

I agree that this file format is not an Apple or Xcode standard (if there were one we would have used it instead obviously), but since there's no such standard we have to pick an arbitrary format for the attribute and for the report. So why not pick this one 🙃

[freak4pc]

I think there are a couple of things that stand out, at least for the MVP of this:

1. It needs to be opt-in to output this JSON, or whatever output that tells you the privacy policy. Possibly a flag on pod install or even a pod inspect privacy (possibly other sub-commands could fall under this inspect subcommand).

2. Filling up this section of the podspec should also be optional or everything would obviously break :) We could allow for specific settings on pod install to warn about pods that don't list privacy details (or even a flag that prohibits them where applicable).

3. This is why I think this "New pods or updates to existing pods would require a value for their privacy details" - is not a great idea. It's the package manager taking a stand and getting in the way instead of helping you. A good PM (IMO) should help you achieve things, but not force you to do things.

[mfaani]

@orta tbh when I approached @AliSoftware I didn't even know there was a fastlane action for this. My thought process was that developers shouldn't have put effort to figure out. Or more accurately, they may not have knowledge of all the internals of the pod hence can't even figure it out. So instead I though pod owners should just be clear about what they do and they do it once. Then every one can use it. I suppose also pod owners could get asked about this, so this makes it easier for them as well.

I see the fastlane integration just as a very juicy and easy to deliver bonus.

Having that said I think @freak4pc makes a good point

"package manager taking a stand and getting in the way instead of helping you" - - is not a great idea

I just think that:

1. The level of effort is to just add a json i.e. it's easy to fix what's broken.
2. Apple is getting in the way. We're just conforming to it 🤷‍♂️. Reducing legal/dev time spent by every app owner.
3. If we don't enforce it then, companies/pods that prefer not to be transparent, would just not opt in.

Perhaps we can make it optional for now then follow the approach that Apple takes and be like:

Starting from 1 May 2021, pods that don't include privacy_details would fail pod lib lint

[freak4pc]

Yeah I would also opt for the more "gradual rollout" approach here. Definitely need to enforce it at some point but perhaps two versions would be a good range.

[AliSoftware]

I like the gradual rollout idea 👍

[freak4pc]

So it seems to me there are two angles, for the two versions:

1. Maintainers: Every call to pod lib lint or pod trunk push should warn if you don't have your privacy set up along with a link to some docs on how to do that. (Perhaps pod repo push as well even though private repos might not care)

2. Consumers: I wouldn't want to generate a huge warning, but maybe an elegant "Some of your dependencies did not provide privacy details: X, Y, Z"

After the first two versions:

1. Maintainers: Every call to pod lib lint or pod trunk push should fail if you don't have your privacy details in your podspec.

2. Consumers: Should still be able to install existing pod specs in trunk without the details, given future ones will be forced to have the details.

WDYT? @prohoney @AliSoftware

[AliSoftware]

Yeah I like that; tho I'm not sure where you're going with the "Consumers" aspects of this.

Consumers would just run pod install and have a new "report" file generated alongside all other files (all the xcconfig files, the sh files etc, …). In that report (the JSON output file presented in this issue description, or another format if we decide to go with something else) we would list all the privacy details gathered from integrated pods… and list the pods that don't provide privacy details info yet (see the unknown_privacy_détails example key provided in the "Backwards Compatibility" section of the proposal).

The consumers would then either never read the generated report and don't care about it, or read it to know that some pods still don't provide their privacy details yet.

Agreed maybe we could consider also printing a warning to stdout upon pod install to let people know about this even if they never open the report file, but that would be a bonus.

Though I like the idea of also having that warning indeed as an additional info about that and to encourage Maintainers to add that info when Consumer will start filing issues about it on their repos when they see the warning.

[freak4pc]

Yeah I like that; tho I'm not sure where you're going with the "Consumers" aspects of this.

Mean this:

Agreed maybe we could consider also printing a warning to stdout upon pod install to let people know about this even if they never open the report file, but that would be a bonus.

Though I like the idea of also having that warning indeed as an additional info about that and to encourage Maintainers to add that info when Consumer will start filing issues about it on their repos when they see the warning.

[freak4pc]

More specifically, if you just output a file (which IMO should be a flag and not the default) without emitting a stdout warning (no exit code of course, just a warning), people will usually not know this is even occurring or an issue.

[AliSoftware]

Yep very good point I like that.

Regarding the generation of the report file, not sure why it wouldn't be the default and would need a flag? What's the harm in generating it in all cases? I fear that putting this being an opt-in flag would make this new feature very hard to discover, with little benefit, right?

[freak4pc]

I'd say that at least an opt-out flag is a must. I'm personally not a fan of a tool generating files for me unless I asked for it explicitly but I can understand the discoverability angle. I don't feel strongly about it :)

[mfaani]

Wouldn't it be easily discoverable given that you're generating a new file — similar to a lock file?

App owners would immediately see it and have to commit it and have to their repo...

[freak4pc]

It's a developer tool but it doesn't mean it shouldn't have nice UI (even if that isn't a GUI).

Generating a file and expecting the end-user to understand they need to do something about this is the wrong way to go IMHO.

If it's auto opt-in I would expect a "Generated privacy details into Pods/Podfile.privacy.json" message along with "Some of your dependencies (X, Y, Z) did not specify privacy details". These things need to be laid out to the consumer.

Two more things about the output file:

1. I don't think it necessarily needs to include the dependencies that don't have privacy (that's why I prefer a warning).
2. Note that many people who run pod install aren't App Owners. Some of them might not figure out what this file is about without a clear stdout message.

[mfaani]

@freak4pc Very good note that this a developer tool and right from the command line things should be apparent. Is the following something we can agree on:

1. Place the report file under /pods/Target Support Files/pods-{targetName}/privacyDetails.json
2. Generation of report file (a.k.a. privacy.json / privacyDetails.json) is defaulted to on. If you don’t want it then you have to opt out with a flag e.g.pod install --no-privacy-details
3. Increase awareness of file creation by emitting an stdout message so folks would know a new file was generated e.g.

Generated privacy details into
/pods/Target Support Files/pods-targetA/privacyDetails.json
/pods/Target Support Files/pods-targetB/privacyDetails.json

4. If some pods don't have privacy_details mentioned in their spec then emit a message to stdout e.g.

WARN | privacy_details: Pods (X,Y,Z) did not specify privacy_details. Starting from 1 Apr 2021. This would be a requirement

5. Add warning when when doing pod install/update changes the privacy details e.g.

WARN | privacyDetails.json changed: One or more pods have updated their privacy details used in targetA

[mfaani]

@freak4pc @orta @AliSoftware I also made a medium size edit onto the issue about how we intend to merge the privacy_details between root specs and sub specs.

[AliSoftware]

This recent article by @k0nserv
can also be useful in brainstorming our feature, providing some more insight into how the feature work in Apple's API. (Especially we can see how the format we propose for the attribute and the report are in fact inspired from Apple's API)

[k0nserv]

Hey 👋🏼

I've attached an example API response(it's for Instagram). This example contains privacy types with the identifiers DATA_USED_TO_TRACK_YOU and DATA_LINKED_TO_YOU. For DATA_USED_TO_TRACK_YOU the data is stored inside dataCategories and for DATA_LINKED_TO_YOU(as well as DATA_NOT_LINKED_TO_YOU) the data is stored in purposes.

This jq query extracts all data types use to track:

jq '.privacy_details.attributes.privacyDetails.privacyTypes[0].dataCategories[].dataTypes[]'  389801252.json 

This one extracts all data types from DATA_LINKED_TO_YOU

jq '.privacy_details.attributes.privacyDetails.privacyTypes[1].purposes[].dataCategories[].dataTypes[]'  389801252.json 

In total there are 32 data types in 14 data categories

JSON Categories and Data Types

{
  "IDENTIFIERS": [
    "Device ID",
    "User ID"
  ],
  "USAGE_DATA": [
    "Advertising Data",
    "Product Interaction",
    "Other Usage Data"
  ],
  "DIAGNOSTICS": [
    "Other Diagnostic Data",
    "Performance Data",
    "Crash Data"
  ],
  "CONTACT_INFO": [
    "Name",
    "Physical Address",
    "Email Address",
    "Phone Number",
    "Other User Contact Info"
  ],
  "PURCHASES": [
    "Purchase History"
  ],
  "LOCATION": [
    "Coarse Location",
    "Precise Location"
  ],
  "USER_CONTENT": [
    "Customer Support",
    "Gameplay Content",
    "Photos or Videos",
    "Emails or Text Messages",
    "Audio Data",
    "Other User Content"
  ],
  "CONTACTS": [
    "Contacts"
  ],
  "OTHER": [
    "Other Data Types"
  ],
  "BROWSING_HISTORY": [
    "Browsing History"
  ],
  "SEARCH_HISTORY": [
    "Search History"
  ],
  "HEALTH_AND_FITNESS": [
    "Fitness",
    "Health"
  ],
  "FINANCIAL_INFO": [
    "Payment Info",
    "Other Financial Info",
    "Credit Info"
  ],
  "SENSITIVE_INFO": [
    "Sensitive Info"
  ]
}

There are 4 privacy types(no data collected, data linked to the user, data not linked to the user, and data used to track the user) and 6 purposes(app functionatlity, analytics, developers advertising, other purposes, product personalization, and third party advertising). The exact identifiers used by Apple's API is in the blog post.

389801252.json.zip

If you have specific questions I can try to help answer them.

EDIT: You can also see examples of the API request and response by opening e.g. Facebook's app store listing on the web and looking at the XHR traffic in the browser developer tools when clicking "See Details" in the "App Privacy" section

EDIT2: Has Apple added support for app privacy details to the App Store Connect API? If so that seems like the best thing to align to

[freak4pc]

@joshdholtz hey man! any ideas if Apple added ASC APIs for the new App Privacy stuff? Is Fastlane already doing some work in that direction?

Thanks <3

[AliSoftware]

@joshdholtz hey man! any ideas if Apple added ASC APIs for the new App Privacy stuff? Is Fastlane already doing some work in that direction?

@freak4pc note that there's a link in the footnote of this PR description to fastlane's own PR adding the feature which already uses the ASC API 😉 This is where we borrowed the inspiration for the podspec attribute actually, borrowing it from the format fastlane uses, which no doubt borrowed it from the ASC API I'm guessing (given that's what they then interact with to automate the submission)? 😛

(Just to reiterate in case this was not clear though: this CocoaPods feature we're proposing here is completely independent of Fastlane and is useful in its own right whatever format we end up using for the attribute and the report file. But since we need to settle on a given format for those anyway, our reasoning was: "why not use the one used by ASC API and by fastlane rather than reinventing a separate format on our own"; that's where the link with fastlane's PR came in the picture)

[joshdholtz]

@AliSoftware @freak4pc The format is something I put together ☺️ It's inspired by the data models that the unofficial ASC API uses for the privacy stuff but simplified a little bit 🤷‍♂️

The privacy endpoints most likely won't ever hit official ASC API ☹️

[mfaani]

@joshdholtz

The privacy endpoints most likely won't ever hit official ASC API ☹️

By 'privacy endpoints' you mean the upload_app_privacy_details_to_app_store?

And what do you mean it won't ever hit official ASC API? You mean even in future if the official ASC API accepts a json then still the privacy endpoints won't be used?

[joshdholtz]

@prohoney The API endpoint that action uses require the Apple ID auth that fastlane has been using for a while. It's not the official App Store Connect API.

The official App Store Connect API is the one that uses the JWT auth. This is the one that Apple has doc for 🙂 Here - https://developer.apple.com/documentation/appstoreconnectapi

The upload privacy details most likely won't get official API support with the JWT auth and will require the Apple ID auth.

I'm not sure if this matters to this PR or not. I've had a newborn in my arms all morning so I haven't been able to look to deeply at this 🤷‍♂️

[mfaani]

@joshdholtz I see. That's very good to know. I actually wasn't aware that there was either an (official or unofficial) ASC API but yeah I don't think it would matter much for this issue.

As to your last paragraph, that's wonderful news. Yet I don't get it, that implies you have 2 extra hands to type with...