Please add llms-full.txt

[7heMech]

What is the type of issue?

No response

What is the issue?

Following up on #17017 bun added llms.txt, but I think we need llms-full.txt for models which don't support sublinks or following links at all.

Where did you find it?

Svelte has llms-full.txt as well.

[MolikoDeveloper]

mm.... llms are not made to download or navigate by default, that is like a "plugin" that you give it to the promt and code...
you can just made a python or js code that extract all the content, with regex that is easy..

why?: is more easy work with individuals files instead of one big file that contain All, even for llm training, you have an structure, sometimes only text, sometimes instructive dataset, etc...

what do i do in this cases:

i give you an example code, you can run it with bun .\index.ts --file .\llms.md --uri "https://bun.sh/*.md" where llms.md is a downloaded file used as base: llms.txt

i hope it helps you

//index.ts
// refactorized with gpt o1
import { readFileSync, writeFileSync, existsSync, mkdirSync } from "fs";
import { resolve } from "path";

interface Options {
    file: string;
    baseURI: string; // optional filter pattern
}

const MARKDOWN_LINK_REGEX = /\[[^\]]*\]\(([^)]+)\)/g;
const ANY_LINK_REGEX = /https?:\/\/[^\s")]+|(?<=\()[^)]+(?=\))/g;

function patternToRegex(pattern: string): RegExp {
    let escaped = pattern.replace(/[.+?^${}()|[\]\\]/g, "\\$&");
    escaped = escaped.replace(/\*/g, ".*");

    const finalRegex = new RegExp(escaped, "i");
    console.log(`[DEBUG] Built pattern => "${pattern}" => Regex:`, finalRegex);
    return finalRegex;
}

/**
 * Checks if a given URL starts with 'http' or 'https'.
 */
function isHTTPURL(url: string): boolean {
    return /^https?:\/\//i.test(url);
}

/**
 * Extracts all Markdown links from the given content and returns them as an array.
 */
function extractMDLinks(content: string): string[] {
    const links: string[] = [];
    let match: RegExpExecArray | null;

    while ((match = MARKDOWN_LINK_REGEX.exec(content)) !== null) {
        links.push(match[1]);
    }
    return links;
}

/**
 * Extracts all links (both HTTP links and any text inside parentheses) from the content.
 */
function extractSubLinks(content: string): string[] {
    const links: string[] = [];
    let match: RegExpExecArray | null;

    while ((match = ANY_LINK_REGEX.exec(content)) !== null) {
        links.push(match[0]);
    }
    return links;
}

/**
 * Processes a URL by downloading (if it's an HTTP URL) or reading (if it's a local file)
 * its content, saving the content to a local file, then recursively processing
 * any additional links found within that content.
 * 
 * If `baseURIRegex` is specified, **only** sub-links that match will be processed.
 */
async function processURL(
    url: string,
    visited: Set<string>,
    baseURIRegex: RegExp | null
): Promise<void> {
    const normalizedUrl = url.trim();

    if (baseURIRegex && !baseURIRegex.test(normalizedUrl)) {
        //console.log(`Skipping link (does not match pattern): ${normalizedUrl}`);
        return;
    }

    if (visited.has(normalizedUrl)) {
        return;
    }
    visited.add(normalizedUrl);

    let content: string;
    try {
        if (isHTTPURL(normalizedUrl)) {
            const response = await fetch(normalizedUrl);
            if (!response.ok) {
                console.error(`Cannot download: ${normalizedUrl} - status code: ${response.status}`);
                return;
            }
            content = await response.text();
        } else {
            // Local file path
            const localPath = resolve(normalizedUrl);
            content = readFileSync(localPath, "utf-8");
        }

        saveContentToFile(normalizedUrl, content);

        const subLinks = extractSubLinks(content);
        for (const subLink of subLinks) {
            await processURL(subLink, visited, baseURIRegex);
        }
    } catch (error) {
        console.error(`Error processing "${normalizedUrl}":`, (error as Error).message);
    }
}

function ensureOutputFolder(): void {
    if (!existsSync("output")) {
        mkdirSync("output");
    }
}

/**
 * Saves content to a file under "output/", using a sanitized version of the URL for the filename.
 */
function saveContentToFile(url: string, content: string): void {
    ensureOutputFolder();
    const baseName = url.replace(/[^a-zA-Z0-9-_]/g, "_").substring(0, 50);
    const outputPath = `output/${baseName}.txt`;
    writeFileSync(outputPath, content, "utf-8");

    console.log(`Saved content from "${url}" to: ${outputPath}`);
}

function parseArgs(): Options {
    const args = process.argv.slice(2);
    const options: Options = { file: "", baseURI: "" };

    for (let i = 0; i < args.length; i++) {
        if (args[i] === "--file" && args[i + 1]) {
            options.file = args[i + 1];
            i++;
        } else if (args[i] === "--uri" && args[i + 1]) {
            options.baseURI = args[i + 1];
            i++;
        }
    }

    if (!options.file) {
        throw new Error('You must specify a file with --file "./main.md"');
    }
    return options;
}

(async function main(): Promise<void> {
    try {
        const { file, baseURI } = parseArgs();
        const initialContent = readFileSync(file, "utf-8");
        const initialLinks = extractMDLinks(initialContent);

        // Build the regex if we have a pattern; otherwise, no filtering
        const baseURIRegex = baseURI ? patternToRegex(baseURI) : null;

        console.log(`[DEBUG] Using baseURI pattern: "${baseURI || "(none)"}"`);
        const visited = new Set<string>();

        for (const link of initialLinks) {
            await processURL(link, visited, baseURIRegex);
        }

        console.log("Process complete. All links and sub-links have been processed.");
    } catch (error) {
        console.error("Error in main:", (error as Error).message);
    }
})();

[MolikoDeveloper]

btw, the first time i do a code like that, it was to extract all content from the url and suburl, convert it to md and train my llm >.< (scrapping)

[7heMech]

@MolikoDeveloper thanks, I didn't know! But still I don't think It'd hurt to have llms-full.txt since it's in the spec.

[7heMech]

I can't find PR or commit, but this has now been implemented. I'll wait for it to come to open source repo to close ig.

[RiskyMH]

The site is closed source and probably will remain like that, so I'll close this as completed!

https://bun.sh/llms-full.txt

[kravetsone]

The site is closed source and probably will remain like that, so I'll close this as completed!

https://bun.sh/llms-full.txt

why it closed source?

[7heMech]

The site is closed source and probably will remain like that, so I'll close this as completed!

https://bun.sh/llms-full.txt

why it closed source?

Maybe they use NodeJS 😜😂