v1.0.4

I am pleased to announce the release of reconcile-tfstate v1.0.4, a critical stability and usability enhancement release. This version addresses crucial operational gaps identified during real-world CI/CD pipeline integration, ensuring more reliable command execution and guaranteeing comprehensive diagnostic reporting even in the face of unexpected errors.

Building on the comprehensive reporting and backup capabilities introduced in previous versions, v1.0.4 makes reconcile-tfstate more resilient and transparent, particularly for automated reconciliation workflows.

---

Key Features & Improvements in v1.0.4:

• Robust Command Execution (Fixes terraform import issues):
  • Corrected Terraform Command Syntax: Resolves a critical bug where terraform import (and terraform state rm) commands generated by the tool were malformed, leading to execution failures. The tool now correctly structures these commands, placing options like -state before positional arguments (ADDR ID).
  • Explicit Terraform Working Directory (--tf-working-dir flag): Introduces a new command-line flag (--tf-working-dir defaulting to .) that explicitly specifies the directory where terraform commands should be executed. This ensures consistency and predictability, especially in complex CI/CD environments where the binary might be invoked from varying paths relative to the Terraform project.
• Guaranteed Error Reporting and Audit Trails to S3:
  • Automatic S3 Upload on Application Crash: Implements a robust error recovery mechanism. If reconcile-tfstate encounters an unexpected error or panic during execution (e.g., a critical terraform command fails), it will now automatically attempt to upload all generated local backups and reports (Markdown and JSON) to your configured S3 bucket. This ensures vital diagnostic information is captured and persisted for debugging, even if the application terminates unexpectedly.
  • Comprehensive Command Execution Logs in Reports: The JSON report (report.<env>.json) now includes a detailed execution_logs array. This array captures the exact command string executed, its stdout, stderr, exit_code, and any error encountered for every remediation command attempted. This provides an invaluable audit trail, clarifying precisely what happened during automated execution.
  • Enhanced JSON Report for Root-Level Errors: The JSON report now features an application_error field at the root level, providing clear visibility into any top-level application panics or critical failures that prevented full report generation.
• Internal Code Quality & Efficiency Enhancements:
  • Optimized Struct Memory Layout: Fields within key Go structs (Config, ResourceStatus, AWSClient, TFStateFile, OutputStateV4, ResourceStateV4, InstanceObjectStateV4, CheckResultsV4, CheckResultsObjectV4, CommandExecutionLog, JSONBackupPaths, JSONResultItem, JSONResults, JSONOutput) have been reordered from largest to smallest type size. This reduces memory padding, leading to a slightly smaller memory footprint and potentially improved performance.
  • Resolved Minor Typos and Logic Gaps: Addressed small but impactful errors such as a uint66 typo, missing Category field references in ResourceStatus, missing Mode field references in ResourceStateV4, and an undefined serviceName variable in a specific ECS verification path.

---

How to Use (New --tf-working-dir flag & Enhanced JSON Output):

To leverage the new working directory control:

# Example in GitHub Actions or any script:
# Assuming your Terraform project root is in: /home/runner/work/another-terraform-project/another-terraform-project/terraform-environments/dev
# And reconcile-tfstate is run from: /home/runner/work/another-terraform-project/another-terraform-project/terraform-environments/dev
./reconcile-tfstate \
  -s3-state "s3://your-bucket/path/to/dev.tfstate" \
  -region "us-west-2" \
  -backups-dir "./backups" \
  --json \
  --should-execute \
  --tf-working-dir "." # Or, if run from a different directory, specify the actual path to your Terraform project

Viewing the Changes and Audit Trails:

• GitHub Actions Artifacts: After any workflow run (even failed ones), navigate to the "Summary" tab of the workflow run in GitHub Actions. Look for the "Artifacts" section. The generated JSON report (tfstate-man-report-env-region.json) will be available for download. This report now contains execution_logs for detailed command outcomes.
• S3 Backups: In the event of a critical failure during execution of the reconcile-tfstate binary itself, look in your S3 bucket (s3://your-bucket/state-backups/YYYY/MM/TIMESTAMP/) for the timestamped report.json, report.txt, and backup state files. These will contain the diagnostic information.

---

Known Limitations / Future Enhancements:

• This tool still only supports Terraform state file format version 4.

This v1.0.4 release significantly enhances reconcile-tfstate's reliability and diagnostic capabilities, making it an even more trustworthy component in your automated infrastructure management workflows.

v1.0.2

I am pleased to announce the release of reconcile-tfstate v1.0.2, a maintenance and enhancement release that further refines its output, improves data integrity reporting, and solidifies its robust backup mechanisms.

Building on the significant resource verification and backup capabilities introduced in v1.0.1, this release focuses on providing more comprehensive and versatile reporting for improved automation and auditing.

Key Features & Improvements in v1.0.2:

• Comprehensive Backup Data in JSON Format:
  • Introduces a new --json flag to render the entire reconciliation report (including configuration, results, and backup details) in a structured JSON format directly to standard output. This enables easy parsing with tools like jq for automation workflows.
  • Always-On File-Based JSON Report: Regardless of whether the --json flag is used for console output, a detailed JSON report (report.<env>.json) and its SHA256 checksum (report.<env>.json.sha256) are now always saved to the designated backup directory (--backups-dir). This ensures a complete, machine-readable audit trail for every run.
• Enhanced State File Backup Integrity Reporting:
  • The new_checksum field in the JSON and Markdown reports is now guaranteed to be populated. It accurately reflects the SHA256 checksum of the post-reconciliation local state file (new.<env>.tfstate), ensuring complete integrity verification even when no terraform import or state rm commands were executed.
  • The new.<env>.tfstate backup file is now always created in the backup directory, mirroring the final state after reconciliation, regardless of whether changes occurred.
• Accurate State File Source Reporting: The state field in the JSON output now correctly reflects the original source URI of the Terraform state file (e.g., s3://bucket/key or /path/to/local.tfstate), providing precise context for state provenance in the report.
• Improved Backup File Naming Consistency: Resolved an issue where backup state files might be saved with redundant extensions (e.g., .tfstate.tfstate). Backup state files are now consistently named original.<env>.tfstate and new.<env>.tfstate.
• Minor Code Refinements: Addressed minor unused parameter warnings and internal logic streamlining for improved code quality and maintainability.

Key Features & Improvements (from v1.0.1, still relevant):

• Expanded Resource Verification Coverage: Dramatically increased the number of AWS resource types fully analyzed and verified against live infrastructure. This includes comprehensive checks for EC2, Auto Scaling, CloudWatch, IAM, Lambda, CloudFront, S3 configurations, and ECS.
• Automated State File Backups with History and Integrity Checks: Introduces the --backups-dir flag (defaulting to ./backups) and automatically generates SHA256 checksums for both original and modified state files. Supports structured S3 backups for --s3-state configurations.
• Improved Reporting: The full reconciliation report, including hashes and change details, is saved to a Markdown file within the backup directory.
• Clearer Console Output: Streamlined the console output for better readability.
• Automated Remediation (--should-execute): Capability to automatically execute suggested terraform import and terraform state rm commands.

How to Use (New --json flag):

# Get JSON output to console, and save all reports to backups/
reconcile-tfstate --s3-state s3://your-bucket/path/to/dev.tfstate --region us-west-2 --json

# Analyze a local state file, execute changes, and save all reports to backups/
reconcile-tfstate --state dev.tfstate --region us-west-2 --should-execute

Known Limitations / Future Enhancements:

• This tool still only supports Terraform state file format version 4.

This v1.0.2 release further enhances reconcile-tfstate's utility, auditability, and ease of integration into automated pipelines.

v1.0.0

I am excited to announce the initial release of reconcile-tfstate, a crucial new tool designed to help us manage and reconcile our Terraform state files with greater confidence and efficiency. This binary was developed to directly address the challenges we've faced with state file discrepancies, particularly after recent multi-region resource movements.

Key Features & Improvements:

• Comprehensive State File Analysis: Scans your Terraform state file (.tfstate) and compares it against live AWS resources in a specified region.
• Intelligent Categorization: Provides a clear, actionable report categorizing every resource into one of six states:
  • OK: Resource exists in both state and AWS, and IDs match.
  • INFO: Local data sources or resources that don't require external AWS verification.
  • WARNING: Resources that are complex to verify fully, or that have minor discrepancies not requiring immediate action (e.g., aws_security_group_rule, aws_region). Manual verification is recommended for these.
  • ERROR: Critical issues found during live AWS checks (e.g., a child resource is present in state, but its parent resource is missing in AWS). These often indicate fundamental infrastructure issues or severe state corruption.
  • REGION_MISMATCH: Resources that exist in the state file but whose ARN indicates a different AWS region than the one being scanned. This helps identify resources that have truly "moved" or are tracked in the wrong state.
  • DANGEROUS: Resources present in the state file but explicitly not found in the target AWS environment. These represent potential drift or unmanaged deletions and require immediate attention.
  • POTENTIAL_IMPORT: Resources found in AWS that exist in your state, but their IDs (or other identifying attributes) don't perfectly match, indicating they might need to be imported to fix the state.
• Automated Remediation Commands: Generates a ready-to-use list of terraform import and terraform state rm commands to quickly synchronize your state file with reality.
• S3 State File Support: Seamlessly downloads and prepares state files directly from S3 for analysis, providing clear instructions for re-uploading modified state (with versioning in mind).

How to Use:

# Analyze a local state file
reconcile-tfstate --state dev.tfstate --region us-west-2

# Analyze an S3-backed state file
reconcile-tfstate --s3-state s3://your-bucket/path/to/dev.tfstate --region us-west-2

Known Limitations / Future Enhancements:

• aws_security_group_rule: Due to the complex, nested nature of security group rules in the AWS API, full attribute-level verification is not yet implemented and they are flagged as WARNING. Manual validation for these specific rules is still recommended.

This tool marks a significant step towards improving our Terraform state hygiene and reducing manual debugging efforts. We encourage everyone working with Terraform to integrate it into their workflow.

Technical Preview

I AM ($IAM) excited to announce the first technical preview pre-release of reconcile-tfstate! This tool helps you detect and manage discrepancies between your Terraform state file and your live AWS infrastructure. Our goal is to provide a robust solution for identifying drift and assisting with state reconciliation.

As a technical preview, this release is intended for early adopters to test and provide feedback. Expect ongoing development, potential breaking changes in future pre-releases, and an evolving feature set. Your input is crucial for shaping the future of this tool!

What's Included

This initial preview focuses on core functionality:

• Terraform State File Parsing: Reads and interprets Terraform state files (currently supporting Terraform state file version 4).
• AWS Resource Verification: Connects to your AWS account to verify the existence and details of resources declared in your state.
• S3 State Support:
  • Download: Automatically downloads Terraform state files from S3 buckets.
  • Upload Instructions: Provides clear AWS CLI commands to upload the modified state back to S3, ensuring versioning is maintained.
• Drift Detection: Identifies resources that are:
  • DANGEROUS: In state but not found in AWS (e.g., manually deleted).
  • POTENTIAL_IMPORT: Found in AWS but with a different ID than in state, suggesting they exist but aren't correctly tracked.
  • REGION_MISMATCH: Resources whose ARNs indicate they are in a different AWS region than the one being checked.
• Actionable Commands: Generates terraform import or terraform state rm commands where applicable, helping you resolve discrepancies.
• Supported Resource Types: Initial support for verifying common AWS resources, including:
  • aws_s3_bucket
  • aws_cloudwatch_log_group
  • aws_key_pair
  • aws_security_group
  • aws_route53_zone
  • aws_lb
  • aws_lb_listener
  • aws_lb_target_group
  • aws_lb_listener_rule
  • Ignores local/data resources like aws_caller_identity, aws_iam_policy_document, archive_file, local_file, and random_password.
• Concurrency Control: Allows you to specify the number of concurrent AWS API calls for faster checks.
• Version Flag: Use --v to display the tool's version.

Known Limitations & Areas for Improvement

• Limited Resource Type Coverage: The current version only supports a subset of AWS resource types. We plan to expand this significantly.
• Read-Only Verification: This tool performs read-only checks against AWS. It does not modify your AWS resources directly. Any suggested terraform commands must be executed manually.
• Error Handling Refinements: Ongoing work to improve error messaging and robustness.
• Terraform State Version Support: Currently restricted to v4 state files.

We Need Your Feedback!

This is your chance to influence the development of reconcile-tfstate. Please report any issues, unexpected behaviors, or feature requests via the GitHub Issues page.

Thank you for being an early tester!