Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Jun 8, 2025

The ContentUploader was previously uploading all articles on every run, regardless of whether the content had changed. This resulted in unnecessary Azure Table Storage operations and inefficient processing.

Changes Made

1. Added Change Detection Infrastructure

  • Added ContentHash field to ContentEntity to store SHA256 hash of article content
  • Implemented CalculateContentHash() method that creates a hash from all content fields (title, category, tags, difficulty, author, description, content)
  • Added GetExistingContent() method to safely retrieve existing content from Azure Table Storage

2. Enhanced Upload Logic

Modified UploadToTableStorage() to:

  • Check if content already exists in storage
  • Compare content hashes to detect changes
  • Only upload if content is new or has been modified
  • Provide clear logging about actions taken:
    • "Skipping {filename} - no changes detected"
    • "Uploaded new content: {filename}"
    • "Updated changed content: {filename}"

3. Improved Console Output

  • Removed redundant success messages from Program.cs
  • Centralized all upload status logging in the upload helper

Example Behavior

Before:

Successfully uploaded tip1.md to Azure Table Storage.
Successfully uploaded tip2.md to Azure Table Storage.
Successfully uploaded tip3.md to Azure Table Storage.

Every file uploaded regardless of changes.

After:

Uploaded new content: tip1.md
Skipping tip2.md - no changes detected
Updated changed content: tip3.md

Only new or changed content is uploaded.

Testing

  • ✅ Verified identical content produces same hash
  • ✅ Verified different content produces different hash
  • ✅ Confirmed existing content detection works correctly
  • ✅ Validated logging provides clear feedback

This optimization significantly reduces Azure Storage operations and improves processing efficiency when running bulk content uploads.

Fixes #3.


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: csharpfritz <78577+csharpfritz@users.noreply.github.com>
Copilot AI changed the title [WIP] ContentUploader should only upload new or changed articles Optimize ContentUploader to only upload new or changed articles Jun 8, 2025
Copilot AI requested a review from csharpfritz June 8, 2025 18:58
@csharpfritz csharpfritz marked this pull request as ready for review June 8, 2025 19:07
@csharpfritz csharpfritz merged commit fb1f93d into main Jun 8, 2025
@csharpfritz csharpfritz deleted the copilot/fix-3 branch June 8, 2025 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ContentUploader should only upload new or changed articles

2 participants