aaa #1
Open
aaa #1
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ntention fixes, documentation improvements. Python DP PLD Accountant - Use sparse epsilons for discrete additive noise mechanisms when using Connect-the-Dots - Standalone method to compute noise parameter for Poisson subsampled Gaussian mechanism - Bug fixes to truncation in PLDPmf Go DP & Privacy on Beam - Update Bazel instructions - Modify the example commands to use the same file names as the README Core DP - Update README with improved explanations Change-Id: I27c9d909810e4d4902ae5fc3b6082087e08324dc GitOrigin-RevId: 630ac5c8457b24f8869c57315ad26a8b52d35ebe
C++: - Fix computation of l2 sensitivity in Gaussian partition selection. This was utility bug, not privacy. The threshold was ~sqrt(max_partition_contributed) times bigger than it can be (i.e. there is significant utility improvements) Go Examples: - Depend on v2.0.0 (instead of v2.0.0-pre2) Privacy on Beam: - Move public partitions-related code to a separate file - Simplify the code a bit by removing redundant abstraction (mapAccum -> pMap) - Remove outdated TODOs - Add tests for partitionMapFn - Use base64-encoding for public partitions serialization - Update dependencies to get rid of security vulnerabilities found by govulncheck Go&Go Examples&Privacy on Beam: - Bump all direct & indirect go & bazel dependencies to latest versions. Accounting: - Make connect-the-dots the default algorithm for constructing PLDs for continuous additive noise mechanisms (Python PLD Accounting) Clustering: - Small cleanups General: - Add missing '.' in README Change-Id: I5912b20a98f2c1d8cf249d41c21c002135264ce9 GitOrigin-RevId: 4f14a0c0bfcf4dd3c616c72724864a0db71092ed
PiperOrigin-RevId: 504483071 Change-Id: Ie4a40fadaa94921554e858e368b103eb5dd787c2 GitOrigin-RevId: 932d7ff43c77a0cc2035ad1a1eb11005341a9436
Change-Id: I7f001d9e592208092d289b7f9a458c17b84b9999 GitOrigin-RevId: 41e19316bf9c0a51ed945136aff871c3b96d9b29
Privacy-on-Beam: - Update dependencies with security vulnerabilities - Remove obsolete TODO PiperOrigin-RevId: 509502492 Change-Id: If8ee2fafea357eaceb68febf4bb4ad8dbddb5a89 GitOrigin-RevId: 16f78c825517cc218ba3585b99efe5b4e81028de
Privacy on Beam: - Unify how Beam universal types are used, refactor aggregations.go & generated_functions.go C++: - Fix errors in the example Java: - Add MechanismType.MECHANISM_NONE. It can be used for implementing mechanisms for testing purposes Change-Id: I1aa1719c2c5201ef857e832387a163fd29ff2bbd GitOrigin-RevId: c1bdb81b90da8c74de1621ffd08b5fe97b8c9973
* removed f'{var=}' for python 3.7 compatibility
* made line lengths fit in 80 characters
Build - Update Protobuf to version 21.12 - Update bazel to version 6.1.1 Python PLD Accounting - Increment minor version of DP accounting library for PyPi release, reflecting improvements in PLD Accountant (Connect-the-Dots) C++ - Extract CalculateThreshold from stddev in GaussianPartitionSelection - Update dependencies - Return absl::Status in statistical tests Pre-thresholding - Release pre-thresholding in C++ and Java Go - Remove deprioritized TODO Change-Id: Ia1f72ef4cca16dcd3ff99e463dc19d7f02a7b0ca GitOrigin-RevId: efdf42fde42fb55e971c7ea38d38092c0fc02d56
C++ DP Lib: * Add GetOutputConfidenceInterval method to BoundedSum Java DP Lib: * Update Java dependencies ZetaSQL Codelab: * Update ZetaSQL dependency * Add GitHub Action build test GitHub Actions: * Enable Run Workflow button in Actions tab Change-Id: I8a26402ae426d0cc3b84031188ec5dd7d5237f85 GitOrigin-RevId: 4620814f3c1baffbcdbfd79dc190b09b7d0abbb2
Java DP Lib: * Switch back to Mockito 4.X ZetaSQL Examples: * Increase timeout for GitHub Action to 60 mins Change-Id: Ifa5cec3e34f8ab522c009e3aea61ef352002bfe7 GitOrigin-RevId: da6af8cdc89e53204f7b6d4dcef7aa67b4446164
Java DP library: - Update Bazel tools for Java: bazel-common and rules_jvm_external Accounting library: - Adds to/from tuple functions for DpEvents. ZetaSQL: - Mention flags to increase available memory for ZetaSQL PiperOrigin-RevId: 522106800 Change-Id: I4db58ab45a0554efee6fd48a30895f868fe4be75 GitOrigin-RevId: 057b69f9b8537c438a872fed7fadce3f11e586ed
Go DP Lib: * Implement mean and variance functions for testing. * Use Go version 1.19 * Update dependencies Privacy-on-Beam: * Use Go version 1.19 * Update dependencies Change-Id: I7f0324ce95ecc0e6422bbaa4b2f97fc31af93f03 GitOrigin-RevId: 350a2920fcc4a4b8310f34d54f40ae7ca1db1fbe
Updated the Go DP Lib dependencies in Privacy-on-Beam and examples. Change-Id: Ibc0ede090e83174cf9fa5029e3a07877b146d31e GitOrigin-RevId: 64490cc9d1c918f4868c041b7a5620963037d25a
C++ DP Library: - [#breaking-change] Remove default L1 sensitivity for Laplace mechanisms. Java DP library: - Add test case for huge long values in count to make sure they will overflow and not clamped to `Long.MAX_VALUE`. DP Accounting library: - Fix some issues with recursively converting DP events into NamedTuple. PiperOrigin-RevId: 524270896 Change-Id: I8bc36d9febfdc93aec799bf080d5724d6bd9fe6f GitOrigin-RevId: b30c1076abab3d16b296d1d1692decfe4cd04bc6
- Removing experimental GetOutputConfidenceInterval method. Java library: - Setting the java version to 11 explicitly. - Remove unnecessary + operators. PiperOrigin-RevId: 527530638 Change-Id: I8024324799ea261a5c3ed10b4509c120c326ea31 GitOrigin-RevId: 2ec19998fc6b7f11772f8caa1a8f2041d70647c0
Java DP lib: - Introduce `LongBoundedSum` an equivalent of `BoundedSum` returning `int64` results. Python accounting library: - Use `typing.List` instead of `list` for Python 3.7 compatibility [Python DP Accounting] - Introduce mu_lower_log_cdf to fix numerical issue in get_delta_for_epsilon [Python PLD Accounting] PiperOrigin-RevId: 528544183 Change-Id: I9baef7232dabaed01c037637a2dc406f51b2cc42 GitOrigin-RevId: 027162b67b04393606408b2f9b4dbf1130385a7a
General changes: - Update Bazel to version 6.2.1 - Update absl dependency for C++ - Add .bazelrc for root workspace - Update Differential Privacy to RE2 release `2023-06-01`. Privacy on Beam: - Switch to the new & preferred "register" package for registering functions & DoFns - Remove reduntant function `intToInt64Fn` - Remove "Fn" from the function names that are not structural DoFns to have consistency - Rename some functions, reorder them to have logical groupings - Remove unused code from testutils. Rename functions, types and fields to be more consistent. - Support arbitrary partition key types in testutils comparison functions. DP accounting library: - Update the `scipy` dependency to `1.9.3`. PiperOrigin-RevId: 535234915 Change-Id: Iccc9088403366ee9c99a621492ddf7a5eee8e8c4 GitOrigin-RevId: 7eb3fc07d923f924a6705d5033380e3255213365
third_party/cephes: - Remove unused imports for inverse_gaussian_cdf_test in Cephes Change-Id: I39eac6837acc83f6dc64fc5135f5e626229483cd GitOrigin-RevId: ab825ecbdfa64bba0b64ce08d3076a1ef00dfdfd
Change-Id: I73636a301a84f1b62a9b3ab972586555528b5559 GitOrigin-RevId: 4a8935a72bcbacdd5d54dd2439eaf2a84412f65b
PiperOrigin-RevId: 543860733 Change-Id: I2c6e8da3eb084a83bad6d2234e02f7ea98687200 GitOrigin-RevId: 272ce4bc50bf8520f7e275b0074ad1974cf6daeb
DP Libs: * Add attack model for the DP Libraries C++ DP Lib: * Update bounded algorithm documentation Go DP Lib: * Add pre-thresholding for PreAggSelectPartitions primitive. * Implement IncrementBy for PreAggSelectPartitions primitive. * Negative counts are now explicitly allowed when incrementing via IncrementBy in Count & PreAggSelectPartitions, meaning it is now possible to decrement privacy ID counts. Java DP Lib: * Verify that low LInf sensitivity leads to noise being added to BoundedMean * Allow negative values in incrementBy in Count and PreAggSelectPartitions. Privacy on Beam: * New API for specifying aggregation and partition selection budgets separately. * Make test mode available outside of the pbeamtest package Change-Id: I877473cf0e62d3feafa802b97171889c58e4a6e4 GitOrigin-RevId: d4488512182b4c9f3f3729447bcc314934de8a7f
Privacy on Beam: * Implement pre-thresholding for Count * Implement pre-thresholding for SelectPartitions Python Accounting Library: * Add docstrings to RDPAccountant methods * Exposes rdp and orders in RdpAccountant to public Change-Id: I07a5aed2fffcde53a1a02b7b217c09acab3fdd26 GitOrigin-RevId: 3f9e1053cc98da3dc511743e3bc062718fab3a53
Privacy on Beam: - Update Go Lib dependency to the latest commit, this fixes the build being broken Go: - More documentation & tests for pre-thresholding in PreAggSelectPartition Java: - Add missing license header to ZeroNoise.java Change-Id: I448925072b118c08f4521258e4ed399039372979 GitOrigin-RevId: 7d05c446cdb9966ce98037447c5664a2d34aa069
Fix reference to `make_event_from_param` in TypeError message
Go: - Implement pre-thresholding for Count - Minor test fixes Privacy on Beam: - Implement pre-thresholding for all remaining aggregations (DistinctPrivacyID, Sum, Mean, Quantiles, DistinctPerKey) - Refactor how parameters are passed on internally in order to increase code reuse & test coverage, reduce unnecessary local variables and shorten function signatures which increases readability. - Minor test fixes C++: - Migrate status macros to absl::Status and absl::StatusOr - Migrate status matchers to absl::Status and absl::StatusOr - Breaking change: remove own fork of status.h and statusor.h - Use absl logging - Remove own logging fork Change-Id: I9b7cb83be4c9418cde1292e335f9e1ab3b418c9c GitOrigin-RevId: 7704385dbfb3dc58f39b3d6816a96645e951386e
Privacy on Beam: - Update to the latest Go Lib dependency Python Accounting: - Update `attrs` in `dp-accounting` to version `~=23.1` Change-Id: I900d787788f76c9c8460348ac55abef187ab2b54 GitOrigin-RevId: 9571367ef35107519c6e715bf07eaee9cf0ebb70
Privacy on Beam: * Merge SelectPartitionsParams & PartitionSelectionParams. Both SelectPartitions and private partition selection of other aggregations now use PartitionSelectionParams. SelectPartitionsParams will be deprecated * Improve documentation of PublicPartitions & PartitionSelectionParams and place them next to each other: clients have to specify one or the other. * Take as input PartitionSelectionDelta instead of PartitionSelectionParams in DistinctPrivacyID PiperOrigin-RevId: 560076986 Change-Id: Id773a5ad7c7a0d3196f9afcd19d27b96aa387f8c GitOrigin-RevId: c75304a91acad3377109c88124c95a32c98a5880
Privacy on Beam: - Add a check for disallowing beam universal types in pbeam.ParDo - Add check for partition selection budget being set when the pre-threshold is set - Rename test mode constants in order to be more readable when called outside the package - Refer to the new privacy budget API in documentation and codelab - fix bugs with the new privacy budget API in DistinctPerKey Python Accounting: - Increment patch version of DP accounting library for PyPi release, reflecting changes in requirements Change-Id: I4cc7eb22625cae4d71fd5dd1098197dfb7b32ea0 GitOrigin-RevId: 9ca4d7d9aa388880d7663979fd0c9acfaf8fb4a1
Java DP Lib: * Amend the `Noise` interface to accept `double` types as `delta` This commit adds an overloaded method addNoise() that accepts double delta value and deprecates the one that accepts @nullable Double. In the next commits we will delete the deprecated methods. This will break the clients who have their own implementation of the Noise interface. They will need to implement the newly added overloadded versions of the addNoise() method. Clients should start calling the new versions of the addNoise() method that accept primitive double delta, because we will delete the deprecated versions. Clients who pass "null" delta value should pass "0" instead. C++ DP Lib: * Do not recalculate Stddev for GaussianMechanism * Add payload to approx bounds output in case of not enough data * Fix error message for invalid pre-thresholding * Fix pre-thresholding implementation * Add `pre_threshold` parameter to existing partition selection classes RDP Accounting: * Use exact formula for the Renyi DP guarantee of Laplace noise addition, instead of an upper bound. Change-Id: I6ef6c15986d5824aa9e08cb795accb4085788eee GitOrigin-RevId: aa095f64e961a7d7c5189956c437d609455de08d
added 21 commits
November 13, 2023 11:41
Overall: - Update Bazel to 6.4.0 C++: - Remove incorrect assumptions about the default capacity of hashtables. With small buffer optimization in swisstable, default capacity is no longer guaranteed to be 0 - Fix a typo in algorithm.h - Fix use of uninitialized value - Update boringssl dependency - Update external absl dependency Java: - Use the new DP lib aggregations API that accepts double delta - Use the new Noise.addNoise() methods that accept double delta. The methods that accept @nullable Double delta were deprecated Change-Id: I289fc97e33c964b78af0013b52c267ba7e73ef52 GitOrigin-RevId: 2da05a4be29a39fc0c55a006365d31c31e93a7f5
Update dependencies for Go, Java, and C++ DP Libs, and for Privacy on Beam. Privacy on Beam: * Implement scalable public partitions for KV types Change-Id: I7b982e1bb1e5640a354ac7ac5a7ed0330be1a913 GitOrigin-RevId: ae90caf4628ab57ec861b51a0db50311ac170ed5
… fixes, deprecation cleanups. General: - Update freshness of documents and fix minor issues. - Remove documentation for deprecated functions Bounded Standard Deviation and Order Statistics. Plume Go: - Add option to CountParams to allow negative outputs to improve data quality of the output. C++ DP: - Fix Float-cast-overflow in quantiles. - Fix import and quantile test dependencies. Java DP: - Update bazel for java to 7.0.0. - Fix truth8 import style. Go DP: - Update go version to 1.21, upgrade Bazel & gazelle versions, upgrade Apache Beam dependency to the latest version. Privacy on Beam: - Mark PrivacySpec constructors of pbeamtest as deprecated. - Add missing doFn registers to fix tests failing with the new Prism runner. Change-Id: Ie08898ee34bdfeb6329701abefd22aa92b1c004a GitOrigin-RevId: 0aa22f05d11bb130df248e842b2ccc6a649c6b00
Privacy on Beam: - Simplify tests by moving error handling to (Approx)Equal functions. - Fix some tests in public_partitions_test.go not being run due to missing ptest.Run calls. Java DP: - Update Rules JVM External to version 6.0 to fix an issue with the `java_export` rule. Go DP: - Update Go version in GitHub actions. Change-Id: If97cda7886ef336b1e6811c9c29e2ca58a0cba7a GitOrigin-RevId: 87b7873d498db4332b0f5b08f80e272be84a1135
Java DP Lib: * Migrate usages of `Truth8.assertThat` to equivalent usages of `Truth.assertThat` Privacy on Beam: * Use ptest.MainWithDefault for pbeamtest similar to pbeam * Deprecate pbeamtest usage for enabling test mode in favor of the new PrivacySpec API. Tests and examples now use the new API * Switch Privacy on Beam tests to the new PrivacySpec API, also includes minor test fixes ZetaSQL Examples: * Update dependencies PiperOrigin-RevId: 605398011 Change-Id: I761849299652918132bad28b67d121de40f8b056 GitOrigin-RevId: cc25bcfe0b9e79d82ae9a07193eaee2bb9e818ba
- Initial release of DP-Auditorium (experimental). - Supports the main testing interfaces and some simple test mechanisms. - Includes a demo with a mean mechanism. PiperOrigin-RevId: 605711565 Change-Id: Ic8da18530eaa3317aff6eb147302cd1bd4122132 GitOrigin-RevId: 8a5d2830fcaa1a85d6e05f972892632a15cb61b9
* Refactor DP Auditorium example * Fix dependencies * Fix README PiperOrigin-RevId: 606329824 Change-Id: I425c7d754416664f710f1e46db338d4241d145ef GitOrigin-RevId: f952488cad35fad5b15fb1306d4b62e5e6a79102
Privacy-on-beam: - Update NewPrivacySpec implementation to match NewPrivacySpecTemp - Remove deprecated pbeamtest constructors in favor of TestMode in PrivacySpecParams - Delete deprecated NewPrivacySpecTemp - Remove remaining logic for handling old privacy spec API - Remove logic for handling old way of specifying budgets PiperOrigin-RevId: 606963656 Change-Id: Iaea63ffd1fa76aaaa486b125d425cb6e64d8b797 GitOrigin-RevId: 84284595c0469a1cead0e74075ecf6227ce3cc4d
PiperOrigin-RevId: 608613104 Change-Id: Id3a9a50ce057352c52c51e90eff71d8f43b8d84a GitOrigin-RevId: ba65bdca42a1722d05f1fb07c8b62f4e8ff22c82
DP Auditorium: - Add dataset generator for classifications tasks - Add dataset generator for Pipeline DP tests - Remove duplicated `VizierGeneratorConfig` Change-Id: Id7e54f8c9c248192f7da035afa86b9ac2f1ecec0 GitOrigin-RevId: 8865b02e81bd6141c9511a38cd5145a598fb46ca
Core DP: - Remove deprecated Element::error proto field. - Remove deprecated ErrorReport::noise_confidence_interval proto field. DP Auditorium: - run_mean_mechanism_example in IPython notebook. Python DP PLD Accountant: - Adds MixtureGaussianPrivacyLoss to privacy_loss_mechanism. - Adds constructor method for discrete PLD of MixtureGaussianPrivacyLoss to privacy_loss_distribution. Privacy on Beam: - Remove remaining references to the oldPrivacySpec API. Change-Id: I10e18421ffe24f6a5ea9a12f692b011b24797e86 GitOrigin-RevId: c4a7436267b393015db2f98560d42c6a8aab3cb7
Python DP PLD: - Update PLD supplementary material with PLD for Mixture of Gaussians mechanism. - Adds case for composing MixtureOfGaussiansDpEvent to PLDAccountant. - Add a DpEvent for a mixture of Gaussians mechanism. Go and Privacy on Beam: - Prepare for v3 release. Privacy on Beam: - Improve error messages for epsilon & delta checks. Change-Id: Ie759b107a8f076d9642b6342da1e6c0c588ccc60 GitOrigin-RevId: 3edd28f6f89660b336a793b2ed27b6b2e006371f
Java DP: - Update Java dependencies for v3.0.0 release. Go DP: - Fix examples version to work with the go tool. Change-Id: If7d39cad87d2f416880712c78a79a2375ca93300 GitOrigin-RevId: 54a8e8833941e617e2d588868e7fd548f6bd81fe
Go DP Library & PoB v3 release: - Update Privacy on Beam dependencies. - Update Go DP examples dependencies. - Update Go core DP dependencies. DP auditorium: - Fix self paths in DP auditorium IPython notebook. Change-Id: I5beb5a18a5fa07712ba59fbfefc32f2984c656de GitOrigin-RevId: d0510b86f2eca9afc2273befe15a3069557be89f
Go DP: - Update Go version in Github setup action for v3.0.0 release. Change-Id: I4dd8892eeb1e150b913bbccc47c6ee43e71fe699 GitOrigin-RevId: ec20cb51cc2bf16e50ca81662041e5dc032a249d
Go Examples & Privacy on Beam - Update examples to latest Go tag. - Update privacy-on-beam to latest Go tag. Change-Id: If95f39ce587b8c0b9e0e2f9e6bb2b9a64d4097e3 GitOrigin-RevId: 2b5046d6b8e1e09cff3d0e45fcab2ebf59359a94
Privacy on Beam: - Update dependencies, including Docker to get security updates - Update pre_thresholding.md to include links to Go Library & PoB Go: - Update README to reflect that pre-thresholding is available in Go Java: - Update the DP library version C++: - Simplify UniformDouble() implementation and make it match its description. DP Accounting: - Fallback to custom bisection search if brentq fails or root does not give sufficient privacy guarantee - Reformatted and cleaned up tests for readability Change-Id: I74d37350c23fde90bd9eadfe7395290176da40b6 GitOrigin-RevId: e4930b09b319e1119eaa480956884a58dac295e0
Privacy on Beam: * Bump golang.org/x/net from 0.22.0 to 0.23.0 DP Auditorium: * Unify RenyiPropertyTester under new divergence tester class * Add interface for divergence based testers * Add example of testing PipelineDP mean mechanism in IPython * Upgrade Histogram tester * Update dependencies DP Accounting: * Increment patch version of DP accounting library for PyPi release Change-Id: I3a513cf7d7c7e144b11c778f792f723dff53132f GitOrigin-RevId: 1833e65df37c76d756d3a62ed6ba99112c0a5dd0
PiperOrigin-RevId: 628482307 Change-Id: If51bed053a057b782c534ced37b13aea1b9bc96d GitOrigin-RevId: b3467a5f002241d940484761cc0b4263885f84cd
We are changing the return type of NumericalMechanism::AddNoise from T to double or int64_t. This reduces confusion when T is an unsigned type. This is *not* a breaking change, since there is an implicit conversion and when using T, which was in the library, and is now in control of the user. Additionally, this commit contains minor clean-ups and an absl dependency update. PiperOrigin-RevId: 634249123 Change-Id: I8423326fb6056e944fbc4d9ceae2c8918edcfedc GitOrigin-RevId: b9193e7d7621855b41d1b430ec01d4534b73f407
Change-Id: Ia6bdcdd058d7aeb8e17e71a20c5c4e86a003c58a GitOrigin-RevId: 549aaa2bf32936824b027b2481d710eb801cb6d8
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
aaaa