fix: capability reporting and scraper auth

[grantdfoster]

🎯 Primary Focus: Capability Detection Fix

The main change in this branch addresses a critical issue where production workers were reporting capabilities they didn't actually have access to.

Key Problem Solved:

• Before: Production used jobServer.GetWorkerCapabilities() which only checked configuration, not real API access
• After: Production now uses capabilities.DetectCapabilities() which actually probes Apify actors to verify real access

---

📊 Change Statistics:

• 9 files changed
• 34 insertions, 102 deletions (net -68 lines - code cleanup!)
• 5 commits spanning from cookie improvements to capability detection fix

---

🔧 Major Changes by Category:

1. Capability Detection Overhaul (Most Critical)

• internal/capabilities/detector.go: Removed JobServer bypass, now always performs real API probing
• internal/jobs/stats/stats.go: Switched from worker-based to real detection
• internal/capabilities/detector_test.go: Updated tests to reflect new behavior

2. Twitter Authentication Improvements

• internal/jobs/twitter/: Multiple files cleaned up and simplified
• internal/jobs/twitter.go: Auth flow improvements
• internal/config/config.go: Configuration enhancements

3. Cookie Management

• internal/jobs/twitter/cookies.go: Cookie handling improvements
• internal/jobs/twitter_test.go: Related test updates

---

🚀 Impact:

Production Behavior Change:

• Before: Silent capability detection based on config only
• After: Visible actor probing logs + accurate capability reporting

Expected Production Logs:

time="..." level=info msg="Running actor trudax~reddit-scraper"
time="..." level=info msg="Actor run started with ID: ..."
time="..." level=info msg="Stopping actor run: ..."
time="..." level=info msg="Updated structured capabilities with real detection: map[...]"

Reliability Improvement:

• ✅ Miners now report only capabilities they actually have
• ✅ Consistent behavior between tests and production
• ✅ No configuration option to bypass real detection

---

🎯 Bottom Line:

This branch fixes a critical reliability issue where miners could claim capabilities they didn't actually have access to. The fix ensures accurate capability reporting by forcing real API probing in production, matching the behavior you see in your tests.

[Copilot]

Pull Request Overview

This PR fixes capability reporting and scraper authentication by removing the skip login verification feature and implementing real capability detection. The changes simplify the Twitter authentication flow and ensure more accurate capability reporting.

• Removed complex skip login verification logic from Twitter scraper authentication
• Simplified Twitter authentication to only use cookie-based authentication without fallback login
• Replaced JobServer-based capability reporting with real capability detection that probes actual APIs and actors

Reviewed Changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
internal/jobs/twitter/scraper.go	Replaced complex login verification logic with simple bearer token setting
internal/jobs/twitter/cookies.go	Removed logout call before loading cookies
internal/jobs/twitter/common.go	Removed skip verification configuration
internal/jobs/twitter/auth.go	Simplified authentication to only support cookie loading, removed login fallback
internal/jobs/twitter.go	Removed skip login verification from auth config
internal/jobs/stats/stats.go	Changed capability reporting to use real detection instead of JobServer
internal/config/config.go	Fixed configuration parsing bugs and improved log level handling
internal/capabilities/detector_test.go	Updated test expectations for real capability detection
internal/capabilities/detector.go	Changed to always perform real capability detection

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[mcamou]

Approving with a couple of coding comments.