Add an initial setup to @sourcegraph/cody-web lib build

[vovakulikov]

Part of sourcegraph/sourcegraph#62448
See sourcegraph/sourcegraph#62792 for more context

Context

We want to reuse the new Cody UI chat and agent API in the sourcegraph web. The first step was done here (where Quinn did the initial research on using code-agent in a web environment through web-worker).

Another important fact is that Cody agent and Cody chat webview from VSCode packages use a few Nodejs APIs (mostly file system calls). Obviously, these APIs are not supported in a browser environment, so at the moment, we replace imports to these APIs with our mocks. This allows us to still build most of the UI for the web but not crush it in browser runtime.

Current state

This PR just prepares a web package to be reusable by sourcegraph/web.

Problems and find outs

• There is a question about which side (consumer or package itself) should compile the @sourcegraph/cody-web package. Should we just publish it in NPM as a plain (original) code, or should @sourcegraph/cody-web provide transpiled code?
• If we publish the original code, that means that the consumer (@sourcegraph/web should know about all possible shim replacements, but neither Vite nor Esbuild allows dynamic aliases on package paths, meaning this won't work

alias: { `vscode`:  '@sourcegraph/cody-agent/src/vscode.shim.ts' }

But we need to shim vscode API since without it, cody-agent is becoming unusable outside of the vscode environment.

Note: esbuild probably can do this with a custom plugin (we did similar thing for resolving web workers in node_modules package resolution here)

• So, this being said, we can try to bundle our code in Cody's repo and publish transpiled code. However, there is a problem with this, too. Neither Vite nor Esbuild can handle web workers when it comes to publishing lib with web-worker in it; they can create a separate entry for them if they see a new Worker() constructor, but they won't include them automatically in the final bundle if we use a library package (which our code-web package is).

• But even if we handle transpiled web worker on the consumer level manually, shims are completely static. This means that the bundler just injects shim code in the entry. Since web worker is a separate entry, this means that shim will be injected twice as a copy (one in the main thread and one in web worker). We can't dynamically communicate between these two shims.

• So comment where I left a suggestion about possible API for file system shim can't be implemented this way (original comment) (basically main thread values won't affect web worker values)

Suggestions

I don't know if it's realistic or not but probably "shim dependencies management" isn't flexible enough sometimes, maybe we could expose important APIs like file systems as runtime dependencies something like

await newAgentClient({
   name: 'cody-server',
   version: '0.1.0',
   workspaceRootUri: 'file:///tmp/cody-server',
   fileSystemSource: new FileSystem()
})

this would mean that we could override this API in web without build-time replacements

If you want to experiment with web-workers and shims replacements, here is a minimal demo version with two packages one of them (lib package) includes web worker logic https://github.com/vovakulikov/web-worker-sandbox

Test plan

• Check that all other packages work as they should be on the high-level

[github-actions]

‼️ Hey @sourcegraph/cody-security, please review this PR carefully as it introduces the usage of an unsafe_ function or abuses PromptString.

[vovakulikov]

@dominiccooney I was looking into supporting different auth options for Cody's agent because while SG instance auth token is a good fit for CLI and IDE clients, it's not a way when we're running agents on the web.

The good news is that since all requests will go from the same domain and will share the same headers with cookies, the standard auth flow should go fine (if I'm not missing anything about Cody network requests, let me know If I am). My idea here is to pass these headers to web-worker where we create code agent and then carry these headers to the place where actual requests happen.

So my question to you — is there a single API client where we do gql request? I've seen that we do have a singleton instance but it seems that we don't pass token or any other configuration trhough constructor there (and we also create a new gql client instance in auth class, where we put a configuration with tokens).

Maybe you know how the client gets the most recent config (is this the place https://sourcegraph.sourcegraph.com/github.com/sourcegraph/cody@030f222bb6e07b7d78059e102e63adb1ff7cf95e/-/blob/vscode/src/main.ts?L189 ?)

Also I'm very open for any other ideas on how to make code agent auth simpler here. I was thinking about providing gql client to the code agent from the consumer (so different consumer will configure their own implementations of client themself) but it seems like a big refactoring work. I'd be glad to hear any other thoughts on this since so far I don't like much any of my ideas on this.

[vovakulikov]

I rebased the most recent changes from the main here #4605 (keep this PR just in case I messed up with rebase conflict resolving), all further work will be done here