Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NuGet pack "The DateTimeOffset specified cannot be converted into a Zip file timestamp" #3793

Merged
merged 25 commits into from Feb 3, 2021
Merged
Show file tree
Hide file tree
Changes from 23 commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
1e0437f
Fix NuGet pack "The DateTimeOffset specified cannot be converted into…
erdembayar Dec 10, 2020
57bb207
Add unit test for Zip file date before 1980 case
erdembayar Dec 10, 2020
e45b889
Address PR review comment by Damon and Nikolche
erdembayar Dec 11, 2020
d7bdb3c
Log that zip file modified dates are set to 1/4/1980 if it's before y…
erdembayar Dec 14, 2020
18e3bf0
Fix failing unit test
erdembayar Dec 14, 2020
3f11fc4
Make modified datetime passed to ZipArchive UTC so any file modified …
erdembayar Dec 16, 2020
5c9d833
Fix typos.
erdembayar Dec 16, 2020
ede2569
Added Zip file 2 second precision limitation comment.
erdembayar Dec 16, 2020
d89fcfb
Merge branch 'dev' into dev-eryondon-CannotBeConverted2ZipfileTimestamp
erdembayar Jan 12, 2021
b212126
Address code review comments by Andy for deterministic packing.
erdembayar Jan 13, 2021
146d4e5
Change to local Datetime for timestamping.
erdembayar Jan 15, 2021
7442f7b
Clean up
erdembayar Jan 29, 2021
dea0eef
Revert accidental changes.
erdembayar Jan 29, 2021
bd1c76a
Cleanup comments.
erdembayar Jan 29, 2021
d9e5479
Address Nikolche's code review.
erdembayar Jan 30, 2021
f24aff3
Address PR comment by Andy.
erdembayar Feb 1, 2021
af38361
Merge branch 'dev' into dev-eryondon-CannotBeConverted2ZipfileTimestamp
erdembayar Feb 1, 2021
69161ae
Address comment about string.Fomatting.
erdembayar Feb 1, 2021
a7f53aa
Fix typo.
erdembayar Feb 1, 2021
c5c3ebf
Fix typo.
erdembayar Feb 1, 2021
49cc7eb
Address more comments for string resouces.
erdembayar Feb 1, 2021
5d812dc
Address additional review comments.
erdembayar Feb 2, 2021
b7737cb
Address string.Format with string.Concat(+) for better performance.
erdembayar Feb 2, 2021
fa51fa0
Address comment by Nikolche.
erdembayar Feb 3, 2021
9bad83e
Check if warningMessage has something before trimming new line.
erdembayar Feb 3, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Expand Up @@ -228,7 +228,7 @@ public Packaging.PackageBuilder CreateBuilder(string basePath, NuGetVersion vers
Path.GetFullPath(Path.GetDirectoryName(TargetPath))), LogLevel.Minimal));
}

builder = new Packaging.PackageBuilder();
builder = new PackageBuilder(false, Logger);

try
{
Expand Down
2 changes: 1 addition & 1 deletion src/NuGet.Core/NuGet.Build.Tasks.Pack/PackTaskLogic.cs
Expand Up @@ -114,7 +114,7 @@ public PackageBuilder GetPackageBuilder(IPackTaskRequest<IMSBuildItem> request)
assetsFilePath));
}

var builder = new PackageBuilder(request.Deterministic)
var builder = new PackageBuilder(request.Deterministic, request.Logger)
{
Id = request.PackageId,
Description = request.Description,
Expand Down
Expand Up @@ -719,15 +719,17 @@ private PackageBuilder CreatePackageBuilderFromNuspec(string path)
path,
_packArgs.GetPropertyValue,
!_packArgs.ExcludeEmptyDirectories,
_packArgs.Deterministic);
_packArgs.Deterministic,
_packArgs.Logger);
}

return new PackageBuilder(
path,
_packArgs.BasePath,
_packArgs.GetPropertyValue,
!_packArgs.ExcludeEmptyDirectories,
_packArgs.Deterministic);
_packArgs.Deterministic,
_packArgs.Logger);
}

private bool BuildFromProjectFile(string path)
Expand Down
5 changes: 5 additions & 0 deletions src/NuGet.Core/NuGet.Common/Errors/NuGetLogCode.cs
Expand Up @@ -872,6 +872,11 @@ public enum NuGetLogCode
/// </summary>
NU5131 = 5131,

/// <summary>
/// File last write timestamp is out of range of what the zip format supports warning
/// </summary>
NU5132 = 5132,

/// <summary>
/// Undefined package warning
/// </summary>
Expand Down
@@ -1,3 +1,4 @@
NuGet.Common.NuGetLogCode.NU5132 = 5132 -> NuGet.Common.NuGetLogCode
NuGet.Common.NuGetLogCode.NU5501 = 5501 -> NuGet.Common.NuGetLogCode
NuGet.Common.NuGetLogCode.NU1012 = 1012 -> NuGet.Common.NuGetLogCode
NuGet.Common.NuGetLogCode.NU5045 = 5045 -> NuGet.Common.NuGetLogCode
Expand Down
@@ -1,3 +1,4 @@
NuGet.Common.NuGetLogCode.NU5132 = 5132 -> NuGet.Common.NuGetLogCode
NuGet.Common.NuGetLogCode.NU5501 = 5501 -> NuGet.Common.NuGetLogCode
NuGet.Common.NuGetLogCode.NU1012 = 1012 -> NuGet.Common.NuGetLogCode
NuGet.Common.NuGetLogCode.NU5045 = 5045 -> NuGet.Common.NuGetLogCode
Expand Down
@@ -1,3 +1,4 @@
NuGet.Common.NuGetLogCode.NU5132 = 5132 -> NuGet.Common.NuGetLogCode
NuGet.Common.NuGetLogCode.NU5501 = 5501 -> NuGet.Common.NuGetLogCode
NuGet.Common.NuGetLogCode.NU1012 = 1012 -> NuGet.Common.NuGetLogCode
NuGet.Common.NuGetLogCode.NU5045 = 5045 -> NuGet.Common.NuGetLogCode
Expand Down
26 changes: 24 additions & 2 deletions src/NuGet.Core/NuGet.Packaging.Extraction/Strings.Designer.cs

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 9 additions & 0 deletions src/NuGet.Core/NuGet.Packaging.Extraction/Strings.resx
Expand Up @@ -865,4 +865,13 @@ Valid from:</comment>
<value>'{0}' cannot be null.</value>
<comment>0 - property name</comment>
</data>
<data name="ZipFileLastWriteTimeStampModifiedMessage" xml:space="preserve">
<value>'{0}' changed from '{1}' to '{2}'</value>
<comment>0 - File name
1 - Original last write timestamp
2 - Updated last write timestamp</comment>
</data>
<data name="ZipFileTimeStampModifiedWarning" xml:space="preserve">
<value>The zip format supports a limited date range. The following files are outside the supported range:</value>
</data>
</root>
Expand Up @@ -10,15 +10,14 @@
using System.IO.Compression;
using System.Linq;
using System.Reflection;
using System.Reflection.Emit;
using System.Text;
using System.Xml.Linq;
using NuGet.Client;
using NuGet.Common;
using NuGet.ContentModel;
using NuGet.Frameworks;
using NuGet.Packaging.Core;
using NuGet.Packaging.PackageCreation.Resources;
using NuGet.Packaging.Rules;
using NuGet.RuntimeModel;
using NuGet.Versioning;

Expand All @@ -31,6 +30,8 @@ public class PackageBuilder : IPackageMetadata
private readonly bool _includeEmptyDirectories;
private readonly bool _deterministic;
private static readonly DateTime ZipFormatMinDate = new DateTime(1980, 1, 1, 0, 0, 0, DateTimeKind.Utc);
private static readonly DateTime ZipFormatMaxDate = new DateTime(2107, 12, 31, 23, 59, 58, DateTimeKind.Utc);
erdembayar marked this conversation as resolved.
Show resolved Hide resolved
private readonly ILogger _logger;
erdembayar marked this conversation as resolved.
Show resolved Hide resolved

/// <summary>
/// Maximum Icon file size: 1 megabyte
Expand All @@ -47,11 +48,22 @@ public PackageBuilder(string path, Func<string, string> propertyProvider, bool i
{
}

public PackageBuilder(string path, Func<string, string> propertyProvider, bool includeEmptyDirectories, bool deterministic, ILogger logger)
: this(path, Path.GetDirectoryName(path), propertyProvider, includeEmptyDirectories, deterministic, logger)
{
}

public PackageBuilder(string path, string basePath, Func<string, string> propertyProvider, bool includeEmptyDirectories)
: this(path, basePath, propertyProvider, includeEmptyDirectories, deterministic: false)
{
}

public PackageBuilder(string path, string basePath, Func<string, string> propertyProvider, bool includeEmptyDirectories, bool deterministic, ILogger logger)
: this(path, basePath, propertyProvider, includeEmptyDirectories, deterministic)
{
_logger = logger;
}

public PackageBuilder(string path, string basePath, Func<string, string> propertyProvider, bool includeEmptyDirectories, bool deterministic)
: this(includeEmptyDirectories, deterministic)
{
Expand Down Expand Up @@ -90,10 +102,21 @@ public PackageBuilder()
{
}

public PackageBuilder(bool deterministic, ILogger logger)
: this(includeEmptyDirectories: false, deterministic: deterministic, logger)
{
}

private PackageBuilder(bool includeEmptyDirectories, bool deterministic)
: this(includeEmptyDirectories: false, deterministic: deterministic, logger: NullLogger.Instance)
{
}

private PackageBuilder(bool includeEmptyDirectories, bool deterministic, ILogger logger)
{
_includeEmptyDirectories = includeEmptyDirectories;
_deterministic = false; // fix in https://github.com/NuGet/Home/issues/8601
_logger = logger;
Files = new Collection<IPackageFile>();
DependencyGroups = new Collection<PackageDependencyGroup>();
FrameworkReferences = new Collection<FrameworkAssemblyReference>();
Expand Down Expand Up @@ -896,10 +919,25 @@ private ZipArchiveEntry CreateEntry(ZipArchive package, string entryName, Compre
return entry;
}

private ZipArchiveEntry CreatePackageFileEntry(ZipArchive package, string entryName, DateTimeOffset timeOffset, CompressionLevel compressionLevel)
private ZipArchiveEntry CreatePackageFileEntry(ZipArchive package, string entryName, DateTimeOffset timeOffset, CompressionLevel compressionLevel, StringBuilder warningMessage)
{
var entry = package.CreateEntry(entryName, compressionLevel);
entry.LastWriteTime = timeOffset;

if (timeOffset.UtcDateTime < ZipFormatMinDate)
{
warningMessage.AppendLine(StringFormatter.ZipFileTimeStampModifiedMessage(entryName, timeOffset.DateTime.ToShortDateString(), ZipFormatMinDate.ToShortDateString()));
entry.LastWriteTime = ZipFormatMinDate;
}
else if (timeOffset.UtcDateTime > ZipFormatMaxDate)
{
warningMessage.AppendLine(StringFormatter.ZipFileTimeStampModifiedMessage(entryName, timeOffset.DateTime.ToShortDateString(), ZipFormatMaxDate.ToShortDateString()));
entry.LastWriteTime = ZipFormatMaxDate;
}
else
{
entry.LastWriteTime = timeOffset.UtcDateTime;
}

return entry;
}

Expand All @@ -921,6 +959,7 @@ private void WriteManifest(ZipArchive package, int minimumManifestVersion, strin
private HashSet<string> WriteFiles(ZipArchive package, HashSet<string> filesWithoutExtensions)
{
var extensions = new HashSet<string>(StringComparer.OrdinalIgnoreCase);
var warningMessage = new StringBuilder();

// Add files that might not come from expanding files on disk
foreach (IPackageFile file in new HashSet<IPackageFile>(Files))
Expand All @@ -933,7 +972,8 @@ private HashSet<string> WriteFiles(ZipArchive package, HashSet<string> filesWith
package,
file.Path,
stream,
lastWriteTime: _deterministic ? ZipFormatMinDate : file.LastWriteTime);
lastWriteTime: _deterministic ? ZipFormatMinDate : file.LastWriteTime,
warningMessage);
var fileExtension = Path.GetExtension(file.Path);

// We have files without extension (e.g. the executables for Nix)
Expand All @@ -957,6 +997,14 @@ private HashSet<string> WriteFiles(ZipArchive package, HashSet<string> filesWith
}
}

warningMessage.Length -= Environment.NewLine.Length;
var warningMessageString = warningMessage.ToString();

if (!string.IsNullOrEmpty(warningMessageString))
{
_logger?.Log(PackagingLogMessage.CreateWarning(StringFormatter.ZipFileTimeStampModifiedWarning(warningMessageString), NuGetLogCode.NU5132));
}

return extensions;
}

Expand Down Expand Up @@ -1079,15 +1127,15 @@ private static void ExcludeFiles(List<PhysicalPackageFile> searchFiles, string b
}
}

private void CreatePart(ZipArchive package, string path, Stream sourceStream, DateTimeOffset lastWriteTime)
private void CreatePart(ZipArchive package, string path, Stream sourceStream, DateTimeOffset lastWriteTime, StringBuilder warningMessage)
{
if (PackageHelper.IsNuspec(path))
{
return;
}

string entryName = CreatePartEntryName(path);
var entry = CreatePackageFileEntry(package, entryName, lastWriteTime, CompressionLevel.Optimal);
var entry = CreatePackageFileEntry(package, entryName, lastWriteTime, CompressionLevel.Optimal, warningMessage);

using (var stream = entry.Open())
{
Expand Down
Expand Up @@ -75,7 +75,7 @@ public static void UpdateFileTimeFromEntry(this ZipArchiveEntry entry, string fi
{
try
{
File.SetLastWriteTimeUtc(fileFullPath, entry.LastWriteTime.UtcDateTime);
File.SetLastWriteTimeUtc(fileFullPath, entry.LastWriteTime.Add(entry.LastWriteTime.Offset).UtcDateTime);
}
catch (ArgumentOutOfRangeException ex)
{
Expand Down
Expand Up @@ -4,6 +4,9 @@ NuGet.Packaging.NupkgMetadataFile.Source.get -> string
NuGet.Packaging.NupkgMetadataFile.Source.set -> void
NuGet.Packaging.PackageBuilder.EmitRequireLicenseAcceptance.get -> bool
NuGet.Packaging.PackageBuilder.EmitRequireLicenseAcceptance.set -> void
NuGet.Packaging.PackageBuilder.PackageBuilder(bool deterministic, NuGet.Common.ILogger logger) -> void
NuGet.Packaging.PackageBuilder.PackageBuilder(string path, System.Func<string, string> propertyProvider, bool includeEmptyDirectories, bool deterministic, NuGet.Common.ILogger logger) -> void
NuGet.Packaging.PackageBuilder.PackageBuilder(string path, string basePath, System.Func<string, string> propertyProvider, bool includeEmptyDirectories, bool deterministic, NuGet.Common.ILogger logger) -> void
static NuGet.Packaging.Rules.AnalysisResources.InvalidUndottedFrameworkInDependencyGroupsWarning.get -> string
static NuGet.Packaging.Rules.AnalysisResources.InvalidUndottedFrameworkInFilesWarning.get -> string
static NuGet.Packaging.Rules.AnalysisResources.InvalidUndottedFrameworkInFrameworkAssemblyGroupsWarning.get -> string
Expand Down
Expand Up @@ -4,6 +4,9 @@ NuGet.Packaging.NupkgMetadataFile.Source.get -> string
NuGet.Packaging.NupkgMetadataFile.Source.set -> void
NuGet.Packaging.PackageBuilder.EmitRequireLicenseAcceptance.get -> bool
NuGet.Packaging.PackageBuilder.EmitRequireLicenseAcceptance.set -> void
NuGet.Packaging.PackageBuilder.PackageBuilder(bool deterministic, NuGet.Common.ILogger logger) -> void
NuGet.Packaging.PackageBuilder.PackageBuilder(string path, System.Func<string, string> propertyProvider, bool includeEmptyDirectories, bool deterministic, NuGet.Common.ILogger logger) -> void
NuGet.Packaging.PackageBuilder.PackageBuilder(string path, string basePath, System.Func<string, string> propertyProvider, bool includeEmptyDirectories, bool deterministic, NuGet.Common.ILogger logger) -> void
static NuGet.Packaging.Rules.AnalysisResources.InvalidUndottedFrameworkInDependencyGroupsWarning.get -> string
static NuGet.Packaging.Rules.AnalysisResources.InvalidUndottedFrameworkInFilesWarning.get -> string
static NuGet.Packaging.Rules.AnalysisResources.InvalidUndottedFrameworkInFrameworkAssemblyGroupsWarning.get -> string
Expand Down
Expand Up @@ -4,6 +4,9 @@ NuGet.Packaging.NupkgMetadataFile.Source.get -> string
NuGet.Packaging.NupkgMetadataFile.Source.set -> void
NuGet.Packaging.PackageBuilder.EmitRequireLicenseAcceptance.get -> bool
NuGet.Packaging.PackageBuilder.EmitRequireLicenseAcceptance.set -> void
NuGet.Packaging.PackageBuilder.PackageBuilder(bool deterministic, NuGet.Common.ILogger logger) -> void
NuGet.Packaging.PackageBuilder.PackageBuilder(string path, System.Func<string, string> propertyProvider, bool includeEmptyDirectories, bool deterministic, NuGet.Common.ILogger logger) -> void
NuGet.Packaging.PackageBuilder.PackageBuilder(string path, string basePath, System.Func<string, string> propertyProvider, bool includeEmptyDirectories, bool deterministic, NuGet.Common.ILogger logger) -> void
static NuGet.Packaging.Rules.AnalysisResources.InvalidUndottedFrameworkInDependencyGroupsWarning.get -> string
static NuGet.Packaging.Rules.AnalysisResources.InvalidUndottedFrameworkInFilesWarning.get -> string
static NuGet.Packaging.Rules.AnalysisResources.InvalidUndottedFrameworkInFrameworkAssemblyGroupsWarning.get -> string
Expand Down
19 changes: 19 additions & 0 deletions src/NuGet.Core/NuGet.Packaging/StringFormatter.cs
@@ -1,6 +1,7 @@
// Copyright (c) .NET Foundation. All rights reserved.
// Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.

using System;
using System.Globalization;

namespace NuGet.Packaging
Expand All @@ -20,5 +21,23 @@ internal static class StringFormatter
source,
contentHash);
}

internal static string ZipFileTimeStampModifiedMessage(
string filePath,
string originalLastWriteTimeStamp,
string updatedLastWriteTimeStamp)
{
return string.Format(CultureInfo.CurrentCulture,
Strings.ZipFileLastWriteTimeStampModifiedMessage,
filePath,
originalLastWriteTimeStamp,
updatedLastWriteTimeStamp);
}

internal static string ZipFileTimeStampModifiedWarning(
string listOfFileTimeStampModifiedMessages)
{
return Strings.ZipFileTimeStampModifiedWarning + Environment.NewLine + listOfFileTimeStampModifiedMessages;
}
}
}
18 changes: 18 additions & 0 deletions src/NuGet.Core/NuGet.Packaging/Strings.Designer.cs

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

9 changes: 9 additions & 0 deletions src/NuGet.Core/NuGet.Packaging/Strings.resx
Expand Up @@ -865,4 +865,13 @@ Valid from:</comment>
<value>'{0}' cannot be null.</value>
<comment>0 - property name</comment>
</data>
<data name="ZipFileLastWriteTimeStampModifiedMessage" xml:space="preserve">
<value>'{0}' changed from '{1}' to '{2}'</value>
<comment>0 - File name
1 - Original last write timestamp
2 - Updated last write timestamp</comment>
</data>
<data name="ZipFileTimeStampModifiedWarning" xml:space="preserve">
<value>The zip format supports a limited date range. The following files are outside the supported range:</value>
</data>
</root>