Update to Newtonsoft 13.0.1 #2504

marmegh · 2022-07-15T18:26:43Z

Bumps Newtonsoft.Json from 10.0.3 and 12.0.3 to 13.0.1. Additionally, this updates JSchema to v1.1.5 and Microsoft.NET.Test.Sdk to 17.4.0-preview-20220707-01 (both of which are implementing updates to Newtonsoft.Json).

Address Advisory: Improper Handling of Exceptional Conditions in Newtonsoft.Json for direct Newtonsoft.Json dependencies.

Supercedes: #2489. #2490, #2491, #2492, #2493, #2494, #2495, #2496, #2497, and #2499.

Release notes

Sourced from Newtonsoft.Json's releases.

13.0.1

New feature - Add JsonSelectSettings with configuration for a regex timeout

Change - Remove portable assemblies from NuGet package

Change - JsonReader and JsonSerializer MaxDepth defaults to 64

Fix - Fixed throwing missing member error on ignored fields

Fix - Fixed various nullable annotations

Fix - Fixed annotations not being copied when tokens are cloned

Fix - Fixed naming strategy not being used when deserializing dictionary enum keys

Fix - Fixed serializing nullable struct dictionaries

Fix - Fixed JsonWriter.WriteToken to allow null with string token

Fix - Fixed missing error when deserializing JToken with a contract type mismatch

Fix - Fixed JTokenWriter when writing comment to an object

12.0.3

New feature - Added support for nullable reference types

New feature - Added KebabCaseNamingStrategy

Change - Package now uses embedded package icon

Fix - Fixed bug when merging JToken with itself

Fix - Fixed performance of calling ICustomTypeDescriptor.GetProperties

Fix - Fixed serializing Enumerable.Empty and empty arrays on .NET Core 3.0

Fix - Fixed deserializing some collection types with constructor

Fix - Fixed deserializing IImmutableSet to ImmutableHashSet instead of ImmutableSortedSet

Fix - Fixed deserializing IImmutableDictionary to ImmutableDictionary instead of ImmutableSortedDictionary

Fix - Fixed deserializing into constructors with more than 256 parameters

Fix - Fixed hang when deserializing JTokenReader with preceding comment

Fix - Fixed JSONPath scanning with nested indexer

Fix - Fixed deserializing incomplete JSON object to JObject

Fix - Fixed using StringEnumConverter with naming strategy and specified values

12.0.2

New feature - Added MissingMemberHandling to JsonObjectAttribute and JsonObjectContract

New feature - Added constructor to JTokenReader to specify initial path

New feature - Added JsonProperty.IsRequiredSpecified

New feature - Added JsonContract.InternalConverter

Change - Moved embedded debug symbols in NuGet package to a symbol package on NuGet.org

Fix - Fixed deserializing nullable struct collections

Fix - Fixed memory link when serializing enums to named values

Fix - Fixed error when setting JsonLoadSettings.DuplicatePropertyNameHandling to Replace

12.0.1

New feature - Added NuGet package signing

New feature - Added Authenticode assembly signing

New feature - Added SourceLink support

New feature - Added constructors to StringEnumConverter for setting AllowIntegerValue

New feature - Added JsonNameTable and JsonTextReader.PropertyNameTable

New feature - Added line information to JsonSerializationException

New feature - Added JObject.Property overload with a StringComparison

New feature - Added JsonMergeSettings.PropertyNameComparison

New feature - Added support for multiple Date constructors with JavaScriptDateTimeConverter

New feature - Added support for strict equals and strict not equals in JSON Path queries

... (truncated)

shaopeng-gh · 2022-07-16T00:00:44Z

src/WorkItems/WorkItems.csproj

    <PackageReference Include="Microsoft.TeamFoundationServer.Client" Version="16.153.0" />
    <PackageReference Include="Microsoft.SourceLink.GitHub" Version="1.0.0" PrivateAssets="All" />
+    <PackageReference Include="Newtonsoft.Json" Version="13.0.1" />


this line looks new in this .csproj, is it needed #ByDesign

Yes, this is needed. Otherwise, TFS will default to an older version of Newtonsoft.

TFS --- do you mean this dll? I see it defaults to 10.0.0

I tried, I think this line is not need, the reason is
Microsoft.Json.Schema 1.1.4 only needs Newtonsoft.Json (>= 9.0.1);
Microsoft.TeamFoundationServer.Client 16.153.0 needs
Newtonsoft.Json (>= 10.0.3);
so the result is 10.0.3;
now this project already using Microsoft.Json.Schema 1.1.5 needs Newtonsoft.Json (>= 13.0.1)
so we will get 13.0.1

Eddy experimented on this one both ways and determined that it is needed at this time.

shaopeng-gh · 2022-07-16T00:06:30Z

src/Test.FunctionalTests.Sarif/Test.FunctionalTests.Sarif.csproj

@@ -47,7 +47,7 @@

  <ItemGroup>
    <PackageReference Include="FluentAssertions" Version="5.10.2" />
-    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.0.0" />
+    <PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.4.0-preview-20220707-01" />


"17.4.0-preview-20220707-01"

I see we are using preview version, looks like we need a certain fix in that version.
can you add a comment in the PR for information purpose. #Closed

This preview version includes the update to Newtonsoft 13.0.1

shaopeng-gh · 2022-07-16T00:18:35Z

src/ReleaseHistory.md

@@ -4,6 +4,7 @@

 * FEATURE: Add `max-file-size-in-kb` argument that allows filtering scan targets by file size. [#2494](https://github.com/microsoft/sarif-sdk/pull/2494)
 * BUGFIX: Fix false positive for `SARIF1002.UrisMustBeValid` for file URIs that omit the `authority`. [#2501](https://github.com/microsoft/sarif-sdk/pull/2501)
+* DEPENDENCY: Updating [Newtonsoft.Json](https://www.nuget.org/packages/Newtonsoft.Json/13.0.1) to v13.0.1, [Microsoft.Json.Schema](https://www.nuget.org/packages/Microsoft.Json.Schema) to v1.1.5, and [Microsoft.Json.Pointer](https://www.nuget.org/packages/Microsoft.Json.Pointer) to v1.1.5 in response to [Advisory: Improper Handling of Exceptional Conditions in Newtonsoft.Json](https://github.com/advisories/GHSA-5crp-9r3c-p9vr). [#2504](https://github.com/microsoft/sarif-sdk/pull/2504)


DEPENDENCY:

previously we have like this, maybe add before the current string?
DEPENDENCY BREAKING: SARIF now requires Newtonsoft.Json v13.0.1. #Resolved

We didn't follow this convention when we updated to v12.0.1. In fact, I don't see a release history note for that specific update. In this case, the release history serves to tell anyone using the SARIF sdk that this version resolves the Newtonsoft vulnerability. I did consider this approach. If during testing, we see any indicators that this is indeed a breaking change, then we can re-evaluate the note.

I need to correct my previous statement. I was able to find the latest release history note and confirmed that this pattern has been repeated and took the suggestion. Thank you.

shaopeng-gh · 2022-07-16T05:40:46Z

src/WorkItems/WorkItems.csproj

@@ -28,10 +28,11 @@
    <PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="3.1.2" />
    <PackageReference Include="Microsoft.Extensions.Logging.ApplicationInsights" Version="2.13.1" />
    <PackageReference Include="Microsoft.Extensions.Logging.Console" Version="3.1.2" />
-    <PackageReference Include="Microsoft.Json.Schema" Version="1.1.4" />
-    <PackageReference Include="Microsoft.Json.Schema.Validation" Version="1.1.4" />


FYI I also see this in solution, not sure if we want to update it as well. It is used in the generation of c# files.

<package id="Microsoft.Json.Schema.ToDotNet" version="1.1.4" #Resolved

src/Sarif/Comparers/ComparerHelpers.cs

…f-sdk into newtonsoftUpgrade

yongyan-gh · 2022-07-18T22:11:30Z

I created a local enlistment of this branch, and run BuildAndTest.cmd and ran BuildAndTest.cmd.
After build was completed, I still saw newtonsoft,json 10.0.3 in the packages folder in source root path.
Looks like some assemblies still reference the old 10.0.3 newtonsoft?

In reply to: 1188373442

yongyan-gh · 2022-07-19T01:14:12Z

don't see where the reference to 10.0.3 comes from, but i got this build error after block the access to 10.0.3

         Acquiring lock for the installation of Newtonsoft.Json 10.0.3
         Acquired lock for the installation of Newtonsoft.Json 10.0.3
         Reading project file C:\repo\github.com\sarifsdkNewton\src\Test.FunctionalTests.Sarif\Test.FunctionalTests.Sarif.csproj.
         Restoring packages for C:\repo\github.com\sarifsdkNewton\src\Test.FunctionalTests.Sarif\Test.FunctionalTests.Sarif.csproj...
         Restoring packages for .NETCoreApp,Version=v3.1...
     1>C:\Program Files\dotnet\sdk\6.0.302\NuGet.targets(130,5): error : Cannot create 'C:\repo\github.com\sarifsdkNewton\packages\newtonsoft.json\10.0.3' because a file or directory with the same name alread
       y exists. [C:\repo\github.com\sarifsdkNewton\src\Sarif.Sdk.sln]
         Resolving conflicts for .NETCoreApp,Version=v3.1...
         System.IO.IOException: Cannot create 'C:\repo\github.com\sarifsdkNewton\packages\newtonsoft.json\10.0.3' because a file or directory with the same name already exists.
            at System.IO.FileSystem.CreateDirectory(String fullPath, Byte[] securityDescriptor)
            at System.IO.Directory.CreateDirectory(String path)
            at NuGet.Packaging.PackageExtractor.<>c__DisplayClass5_0.<<InstallFromSourceAsync>b__0>d.MoveNext()
         --- End of stack trace from previous location ---

In reply to: 1188373442

EasyRhinoMSFT

…f-sdk into newtonsoftUpgrade

revoking review

EasyRhinoMSFT · 2022-07-19T17:48:25Z

I see the 10.0.3 dependency in these packages:
microsoft.applicationinsights.windowsserver.telemetrychannel
microsoft.azure.kusto.data
microsoft.visualstudio.services.client
microsoft.teamfoundationserver.client

We use Microsoft.Extensions.Logging.ApplicationInsights, where all versions are marked deprecated except the latest. This is where the ref to microsoft.applicationinsights.windowsserver.telemetrychannel comes from I believe.

In reply to: 1188488582

marmegh · 2022-07-20T05:55:05Z

Just wanted to note that as of the updates to the above packages, the scan from the build pipeline is passing.

In reply to: 1189381557

shaopeng-gh

eddynaka · 2022-07-20T06:04:51Z

src/build.props

@@ -10,8 +10,8 @@
    <Company Condition=" '$(Company)' == '' ">Microsoft</Company>
    <Product Condition=" '$(Product)' == '' ">Microsoft SARIF SDK</Product>
    <Copyright Condition=" '$(Copyright)' == '' ">© Microsoft Corporation. All rights reserved.</Copyright>
-    <VersionPrefix>2.4.15</VersionPrefix>
-    <PreviousVersionPrefix>2.4.14</PreviousVersionPrefix>
+    <VersionPrefix>2.4.16</VersionPrefix>


16

If we are changing this, we should also update the release history header #Closed

eddynaka · 2022-07-20T16:30:50Z

src/ReleaseHistory.md

@@ -4,6 +4,7 @@

 * FEATURE: Add `max-file-size-in-kb` argument that allows filtering scan targets by file size. [#2494](https://github.com/microsoft/sarif-sdk/pull/2494)
 * BUGFIX: Fix false positive for `SARIF1002.UrisMustBeValid` for file URIs that omit the `authority`. [#2501](https://github.com/microsoft/sarif-sdk/pull/2501)
+* DEPENDENCY BREAKING: SARIF now requires Newtonsoft.JSON 13.0.1. Updating [Newtonsoft.Json](https://www.nuget.org/packages/Newtonsoft.Json/13.0.1) to v13.0.1, [Microsoft.Json.Schema](https://www.nuget.org/packages/Microsoft.Json.Schema) to v1.1.5, and [Microsoft.Json.Pointer](https://www.nuget.org/packages/Microsoft.Json.Pointer) to v1.1.5 in response to [Advisory: Improper Handling of Exceptional Conditions in Newtonsoft.Json](https://github.com/advisories/GHSA-5crp-9r3c-p9vr). [#2504](https://github.com/microsoft/sarif-sdk/pull/2504)


Newtonsoft.JSON 13.0.1

we updated a few more dependencies. don't we have to include those as well? #Resolved

eddynaka

EasyRhinoMSFT

yongyan-gh

marmegh and others added 7 commits July 5, 2022 16:37

Update to Newtonsoft 13.0.1

3392e26

Updating packages

e397e1e

Adding newtonsoft to workitems until jschema is fixed.

97680df

Using preview version (which contains newtonsoft fix)

216d72f

Updating JSchema nuget packages.

fbaaf18

Updating release history

f20fb9d

Resolved merge conflicts with main

26a538b

shaopeng-gh reviewed Jul 16, 2022

View reviewed changes

Updating another newtonsoft reference.

3d143a6

shaopeng-gh reviewed Jul 16, 2022

View reviewed changes

github-advanced-security bot found potential problems Jul 17, 2022

View reviewed changes

src/Sarif/Comparers/ComparerHelpers.cs Fixed Show fixed Hide fixed

eddynaka added 3 commits July 18, 2022 09:07

Updating packages

e6bd7dd

Merge branch 'newtonsoftUpgrade' of https://github.com/microsoft/sari…

d077083

…f-sdk into newtonsoftUpgrade

Updating autogenerated files

c640726

EasyRhinoMSFT previously approved these changes Jul 19, 2022

View reviewed changes

marmegh added 2 commits July 19, 2022 09:35

Jschema packages.config reference update

b663d1a

Merge branch 'newtonsoftUpgrade' of https://github.com/microsoft/sari…

968870f

…f-sdk into newtonsoftUpgrade

More package updates + increment version #

bf3369c

marmegh marked this pull request as ready for review July 20, 2022 05:50

marmegh requested review from eddynaka and michaelcfanning as code owners July 20, 2022 05:50

shaopeng-gh approved these changes Jul 20, 2022

View reviewed changes

eddynaka reviewed Jul 20, 2022

View reviewed changes

Revert version increment to roll into single release PR.

ef08a64

eddynaka reviewed Jul 20, 2022

View reviewed changes

eddynaka approved these changes Jul 20, 2022

View reviewed changes

EasyRhinoMSFT approved these changes Jul 20, 2022

View reviewed changes

Updating release history for additional package updates.

23d1f21

yongyan-gh approved these changes Jul 20, 2022

View reviewed changes

marmegh merged commit 41c12ef into main Jul 21, 2022

marmegh deleted the newtonsoftUpgrade branch July 21, 2022 15:46

marmegh mentioned this pull request Jul 21, 2022

Creating release v.2.4.16 #2507

Merged

marmegh mentioned this pull request Aug 4, 2022

Newtonsoft and Layout Changes microsoft/sarif-website#142

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update to Newtonsoft 13.0.1 #2504

Update to Newtonsoft 13.0.1 #2504

marmegh commented Jul 15, 2022 •

edited

Loading

shaopeng-gh Jul 16, 2022 •

edited by marmegh

Loading

marmegh Jul 16, 2022

shaopeng-gh Jul 16, 2022

shaopeng-gh Jul 16, 2022

marmegh Jul 20, 2022

shaopeng-gh Jul 16, 2022 •

edited

Loading

marmegh Jul 16, 2022

shaopeng-gh Jul 16, 2022 •

edited by marmegh

Loading

marmegh Jul 16, 2022

marmegh Jul 20, 2022

shaopeng-gh Jul 16, 2022 •

edited by marmegh

Loading

yongyan-gh commented Jul 18, 2022 •

edited by marmegh

Loading

yongyan-gh commented Jul 19, 2022

EasyRhinoMSFT left a comment

EasyRhinoMSFT commented Jul 19, 2022

marmegh commented Jul 20, 2022

shaopeng-gh left a comment

eddynaka Jul 20, 2022 •

edited by marmegh

Loading

eddynaka Jul 20, 2022 •

edited by marmegh

Loading

eddynaka left a comment

EasyRhinoMSFT left a comment

yongyan-gh left a comment

Update to Newtonsoft 13.0.1 #2504

Update to Newtonsoft 13.0.1 #2504

Conversation

marmegh commented Jul 15, 2022 • edited Loading

13.0.1

12.0.3

12.0.2

12.0.1

shaopeng-gh Jul 16, 2022 • edited by marmegh Loading

Choose a reason for hiding this comment

marmegh Jul 16, 2022

Choose a reason for hiding this comment

shaopeng-gh Jul 16, 2022

Choose a reason for hiding this comment

shaopeng-gh Jul 16, 2022

Choose a reason for hiding this comment

marmegh Jul 20, 2022

Choose a reason for hiding this comment

shaopeng-gh Jul 16, 2022 • edited Loading

Choose a reason for hiding this comment

marmegh Jul 16, 2022

Choose a reason for hiding this comment

shaopeng-gh Jul 16, 2022 • edited by marmegh Loading

Choose a reason for hiding this comment

marmegh Jul 16, 2022

Choose a reason for hiding this comment

marmegh Jul 20, 2022

Choose a reason for hiding this comment

shaopeng-gh Jul 16, 2022 • edited by marmegh Loading

Choose a reason for hiding this comment

yongyan-gh commented Jul 18, 2022 • edited by marmegh Loading

yongyan-gh commented Jul 19, 2022

EasyRhinoMSFT left a comment

Choose a reason for hiding this comment

EasyRhinoMSFT commented Jul 19, 2022

marmegh commented Jul 20, 2022

shaopeng-gh left a comment

Choose a reason for hiding this comment

eddynaka Jul 20, 2022 • edited by marmegh Loading

Choose a reason for hiding this comment

eddynaka Jul 20, 2022 • edited by marmegh Loading

Choose a reason for hiding this comment

eddynaka left a comment

Choose a reason for hiding this comment

EasyRhinoMSFT left a comment

Choose a reason for hiding this comment

yongyan-gh left a comment

Choose a reason for hiding this comment

marmegh commented Jul 15, 2022 •

edited

Loading

shaopeng-gh Jul 16, 2022 •

edited by marmegh

Loading

shaopeng-gh Jul 16, 2022 •

edited

Loading

shaopeng-gh Jul 16, 2022 •

edited by marmegh

Loading

shaopeng-gh Jul 16, 2022 •

edited by marmegh

Loading

yongyan-gh commented Jul 18, 2022 •

edited by marmegh

Loading

eddynaka Jul 20, 2022 •

edited by marmegh

Loading

eddynaka Jul 20, 2022 •

edited by marmegh

Loading