feat: migrate types

[grantdfoster]

High-Level Summary of tee-worker PR Changes

Major Architectural Changes

1. TEE Types Migration - The most significant change is the complete migration and consolidation of the tee-types repository into tee-worker. This involved:

   • Moving all type definitions from the separate tee-types package into tee-worker/api/types/
   • Updating all import statements throughout the codebase to use local types instead of external tee-types
   • Consolidating job definitions, argument types, and result structures

2. Job System Refactoring - Complete overhaul of the job handling system:

   • Introduced a new JobType system with proper validation
   • Added comprehensive job argument unmarshalling with type-specific validation
   • Implemented capability-based job routing and validation
   • Added support for multiple job types: LinkedIn, Reddit, TikTok, Twitter, Web scraping, and LLM operations

New Features Added

3. LinkedIn Integration - Full LinkedIn scraping capabilities:

   • Profile fetching with comprehensive data structures
   • Industry, experience, and seniority type definitions
   • LinkedIn-specific job arguments and validation

4. Enhanced Scraping Support:

   • Reddit: Post and comment URL validation, search capabilities
   • TikTok: Search by query and trending functionality
   • Twitter: Enhanced search with profile results
   • Web: General web scraping with depth control
   • LLM: Language model integration capabilities

5. Utility Functions - Added new utility packages:

   • Math utilities (Min, Max functions)
   • Set operations for capability management
   • Enhanced argument validation and unmarshalling

Code Quality Improvements

6. Testing Infrastructure:

   • Comprehensive test suites for all new job types
   • Unit tests for utility functions
   • Integration tests for API endpoints
   • Updated GitHub Actions workflows

7. Configuration and Development:

   • Added collaboration rules for development workflow
   • Updated IDE configuration files
   • Enhanced dependency management with Dependabot
   • Improved CI/CD pipeline with better test coverage

API and Client Changes

8. Client Library Updates:

   • Updated HTTP client to use local types instead of external tee-types
   • Improved error handling and response processing
   • Enhanced job signature generation and validation

9. Internal API Refactoring:

   • Updated route handlers to use new job type system
   • Improved capability detection and reporting
   • Enhanced job processing pipeline

Key Technical Details

• 88 files changed with 5,168 additions and 923 deletions
• Complete removal of external tee-types dependency
• Introduction of type-safe job argument handling
• Enhanced validation and error reporting throughout the system
• Improved modularity with better separation of concerns

This PR represents a major consolidation effort that brings all TEE-related types and functionality under one repository while significantly expanding the scraping capabilities and improving the overall architecture of the system.

[Copilot]

Pull Request Overview

This PR migrates all type definitions from the external tee-types package into the local tee-worker/api/types/ directory, consolidating the codebase and eliminating the external dependency. The migration includes comprehensive job argument validation, capability-based job routing, and enhanced support for multiple scraping platforms.

Key changes:

• Complete removal of external tee-types dependency and migration of all types to local api/ directory
• Implementation of type-safe job argument unmarshalling with comprehensive validation
• Addition of utility functions for mathematical operations and set operations

Reviewed Changes

Copilot reviewed 83 out of 89 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
pkg/util/set.go	Implements generic Set data structure with union, intersection, and difference operations
pkg/util/math.go	Adds Min and Max utility functions for ordered types
api/types/*.go	Consolidates all type definitions from external tee-types package
api/args/*.go	Implements type-safe job argument structures with validation
internal/jobs/*.go	Updates all job handlers to use local types instead of external tee-types
pkg/client/http.go	Updates HTTP client to use local types

Files not reviewed (4)

• .idea/.gitignore: Language not supported
• .idea/modules.xml: Language not supported
• .idea/tee-types.iml: Language not supported
• .idea/vcs.xml: Language not supported

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[rapidfix]

Oh yes!

[grantdfoster]

I think there is one small thing to look at here - how the tee indexer imports the tee client. not 100% sure it needs the ego import, investigating...

[grantdfoster]

It was just this piece importing tee / ego, so duplicated in client code

type EncryptedRequest struct {
	EncryptedResult  string `json:"encrypted_result"`
	EncryptedRequest string `json:"encrypted_request"`
}

[rapidfix]

It was just this piece importing tee / ego, so duplicated in client code

type EncryptedRequest struct {
	EncryptedResult  string `json:"encrypted_result"`
	EncryptedRequest string `json:"encrypted_request"`
}

can't we move it from the worker in a place that doesn't have ego imports?