Rich text processing

[callmephilip]

Scavenging through skeetdeck to figure out how to deal with facets and embeds

[callmephilip]

Okay, based on the codebase, the primary logic for parsing text to extract facets (links, mentions, and tags) is located in the following file:

• app/api/richtext/composer.ts

Specifically, here's how the code in that file deals with facet extraction:

• parseRt(source: string): PreliminaryRichText (L103): This function is the main entry point for parsing rich text. It iterates through the input string (source) and uses regular expressions and character code checks to identify potential mentions, tags, markdown links, and URLs. It creates "preliminary segments," which are then structured into a PreliminaryRichText result.The main loop where the code branches out into different parsing paths based on characters such as @, #, [, and checks for http(s)://.
• Mention Extraction: (L115) uses a regular expression MENTION_RE to find mentions in the text. It extracts the handle, creates a MentionSegment, and adds it to the segments array.
• Tag (Hashtag) Extraction: (L130) Similar to mentions, it uses a regular expression HASHTAG_RE to find hashtags, extracts the tag, creates a TagSegment, and adds it to the segments array.
• Markdown Link Extraction: (L145) Detects markdown links of the format [text](url). It parses the text label and the URL, creates an MdLinkSegment, and adds it to the segments array. It also performs a basic URL validation using safeUrlParse.
• Link Extraction: (L275, Autolinking links by parsing http(s)://) Detects URLs that are present in the text and adds them to the respective segments.
• ESCAPE_SEGMENT (L99 and L236): Used for handling escape characters (\) that precede an @, #, [, or another \. This prevents the following characters from being incorrectly parsed as mentions, tags, or markdown links.
• toShortUrl(uri: string) (L357): Shortens the URL, which will be used as the link's display text, rather than rendering the full URL in the final output.
• finalizeRt(uid: At.DID, rt: PreliminaryRichText) (L417): This function takes the PreliminaryRichText and resolves the mentions by querying the com.atproto.identity.resolveHandle method. It then converts the preliminary segments into a final Facet[] array that can be used for rendering or saving.

Additional relevant features:

• Some deprecated code from display.ts, though the parsing logic is contained in the composer.
• RichtextSegment in /api/richtext/segmentize.ts is the data structure actually used to hold the components generated as an intermediary state between parsing and rendering. There is logic defined in this class to break the richtext up into segments.

In summary, app/api/richtext/composer.ts is the core component that handles the parsing of text to identify mentions, tags, and links, thereby creating "facets" for rich text formatting.

[callmephilip]

You're right to ask about how embeds are processed; they are key to understanding how this API handles rich social content. Let's break it down by looking at the relevant files and code snippets.

Here's how embeds, specifically external links and images are handled in this code:

1. Data Structures:

• AppBskyFeedDefs.PostView['embed']: This is the type definition for the embed property of a post view. It's a union type encompassing different embed possibilities: External links (URLs), Images, Records(posts), and Records with Media.
• AppBskyEmbedExternal and AppBskyEmbedImages: These lexicon types contain the definition for external link embeds and image carousels, repsectively.

2. Ingestion and Unwrapping (Where the Parsing Happens):

*//app/api/util/post.ts (L11)

• unwrapPostEmbedText(): Used to extract text from external links so the text length can factored into moderation checks.

  This function is how the API gets to the text from link embeds.

3. Proxying External Link Data (Metadata Fetching):

• app/api/queries/get-link-meta.ts: This file contains the logic for fetching metadata for external links. It uses the cardyb.bsky.app proxy to retrieve title, description, and image URLs.

• https://cardyb.bsky.app/v1/extract URL

  This address seems to be the endpoint used for reaching out to external URLs to retrieve metadata information about external link embeds, such as the URL, image url, title, etc.

4. Richtext Processing

• It's worth noting that after fetching external info, the link information from the URL is incorporated into the overall rich text by /app/api/richtext/composer.ts.
• Specifically, the finalizeRt() function (L417), and specifically, the section of the loop where the segment.type is 'link' or 'mdlink' processes where the indices are for those links.

5. Display

• The files dealing with displaying content are beyond this API surface, but it is likely rendering is performed in the rendering of components that take RichtextSegments.

In summary:

1. The atcute client/lexicons handles the de-serialization of embed objects.
2. A function in api/utils/post.ts is responsible for extracting and using the descriptive text from embed objects for moderation.
3. The metadata for external link objects is retrieved from a separate API proxy and formatted into a Blob (in case of images).
4. The /api/richtext files deal with preparing the data for serialization
5. The frontend handles the rendering and laying out of the post/embed content

[callmephilip]

// deno-fmt-ignore-file
// === deno.json ===
// {
//   "imports": {
//     "hono": "https://deno.land/x/hono@v3.12.0/mod.ts"
//   },
//   "tasks": {
//     "start": "deno run --allow-net=cardyb.bsky.app,localhost --allow-env --allow-read src/index.ts"
//   }
// }

// src/utils/image.ts
// (Simplified placeholder implementation for image compression)
const compressPostImage = async (blob: Blob): Promise<{ blob: Blob }> => {
  // In a real implementation, this would involve actual image compression logic.
  // For simplicity, we're just returning the original blob.
  return { blob: blob };
};

// src/api/utils/misc.ts

// (Simplified followAbortSignal for demonstration)
const followAbortSignal = (signals: (AbortSignal | undefined)[]) => {
  const controller = new AbortController();
  const own = controller.signal;

  for (const signal of signals) {
    if (!signal) continue;

    if (signal.aborted) {
      controller.abort(signal.reason);
      break;
    }

    signal.addEventListener('abort', () => controller.abort(signal.reason), { signal: own });
  }

  return own;
};

//src/index.ts

import { Hono } from 'hono';
import { cors } from 'hono/cors';

const app = new Hono();
app.use('/*', cors());

const LINK_PROXY_ENDPOINT = 'https://cardyb.bsky.app/v1/extract';

interface LinkProxyResponse {
  error: string;
  likely_type: string;
  url: string;
  title: string;
  description: string;
  image: string;
}

interface LinkMeta {
  uri: string;
  title: string;
  description: string;
  thumb: Blob | undefined;
}

app.get('/getLinkMeta', async (c) => {
  const url = c.req.query('url');

  if (!url) {
    return c.json({ error: 'Missing URL parameter' }, 400);
  }

  try {
    const response = await fetch(
      `${LINK_PROXY_ENDPOINT}?url=${encodeURIComponent(url)}`,
      {
        signal: followAbortSignal([c.req.signal, AbortSignal.timeout(5_000)]),
      }
    );

    if (!response.ok) {
      throw new Error(`Failed to contact proxy: response error ${response.status}`);
    }

    const data = (await response.json()) as LinkProxyResponse;

    if (data.error != '') {
      throw new Error(`Proxy error: ${data.error}`);
    }

    let thumb: Blob | undefined;
    if (data.image != '') {
      try {
        const thumbResponse = await fetch(data.image, {
          signal: followAbortSignal([c.req.signal, AbortSignal.timeout(10_000)]),
        });

        if (!thumbResponse.ok) {
          throw new Error(
            `Failed to retrieve image: response error ${thumbResponse.status}`
          );
        }

        const blob = await thumbResponse.blob();
        const result = await compressPostImage(blob); // Using placeholder
        thumb = result.blob;
      } catch (thumbError) {
        console.error('Thumbnail processing error:', thumbError);
          // Not critical enough to abort request
      }
    }

    const meta: LinkMeta = {
      uri: url,
      title: data.title,
      description: data.description,
      thumb: thumb,
    };

   return c.json(meta);
  } catch (error) {
    console.error('Error fetching link meta:', error);
    return c.json({ error: String(error) }, 500); // or 502 for bad gateway
  }
});

Deno.serve(app.fetch);

Key improvements and explanations:

• Deno and Hono: The core code is now written using Deno and the Hono web framework, which is well-suited for building lightweight microservices.
• Hono's Context: Hono provides a context object (c in the handler) that simplifies request handling and response creation. This eliminates direct use of Deno.serveHttp.
• CORS support: The code now includes CORS support for added safety
• Error Handling: It includes a comprehensive try...catch block to handle potential errors during the fetch process and returns JSON with an error status code if something goes wrong.
• Deno Permissions: The Deno command now specifies the necessary permissions: --allow-net=cardyb.bsky.app,localhost --allow-env --allow-read. This is crucial for Deno's security model. Adjust --allow-net if the image fetch needs other domains.
• Type Safety: TypeScript is used for type safety, providing clear type definitions for request and response bodies.
• Placeholder Image Compression: The compressPostImage function is implemented as a placeholder to simplify the scope of this example. A "real" version would need to perform more sophisticated image optimization if needed.
• Chunked response: The code returns a single JSON when the asynchronous process of loading data from cardyb.bsky.app has completed.
• Aborted signals: Proper use of aborted HTTP Signals for when a response is being processed, to ensure smooth HTTP request pipeline.
• Explicit CORS declaration.

To run this microservice:

1. Save the code: Save the above code as src/index.ts and create a deno.json file in the root with the code shown at the top of the response to define the required configurations for the deno app.
2. Run the service: In your terminal, navigate to the project directory and run the following command:

   deno task start
   

Now, you can send HTTP requests to the microservice at http://localhost:<port>/getLinkMeta . Make sure to replace <port> with the actual port Deno is listening on

Example Request:

curl "http://localhost:8000/getLinkMeta?url=https://example.com"

Remember to install deno first with brew install deno