20250721 ETL Notes

july212005: some thoughts about how we are supposed to deal with all of this traffic from many providers:

• we have a bunch of fixtures/maps describing various remote endpoints
• we have some concept of 'types' of traffic ( socials, email, messages, location, receipts ) -> presumably there are hypersigils in the user experience that iconify each of these types
• we can use a codeless approach or a code approach, but a code approach does require writing probably a ton of similar variations of handlers
• what does the pipeline look like?
  1. there are a bunch predefined providers
  2. users define connections
  3. make actual connection - we use providers to establish connections
  4. cron runs on a connection on a schedule
  5. [todo] we collect raw data and store it - mark and sweep can digest later
  6. [todo] then we manufacture companion intelligence memory objects of the formal types we've defined elsewhere, from this data, this can be a separate operation we have to define what our archtypical kinds of artifacts are ->

// some kind of concept of a type of artifact for user level and for our system as a whole to categorize / normalize the types const myartifacttypes = { "messages": { description: "these are private messages a user receives", art: "some artwork that is used to represent the primitive", } }

interface connection_handler { fetch() }

// presumably these connections are crons - although probably not this granular, there is probably some kind of factory that finds the right handler for each source @cron handlerlogic_pass_1_for_a_connection = () => { read from a source switch(connection.type) { case google: fetch only new state store raw state somewhere case microsoft: fetch only new state store raw state somewhere } }

@cron handlerlogic_for_google_pass2 = () => { look for undigested traffic (mark and sweep) make sure it is not a dupe promote it based on artifact type }

note:

• manufactured objects should have provenance - refering to raw data (which we keep)
• ... now we have this actionable memory in theory ... we can reason about
• we may want exercises / test data / tests and so on