Skip to content

Commit

Permalink
Merge remote-tracking branch 'origin/master' into limit-free
Browse files Browse the repository at this point in the history
  • Loading branch information
haraldschilly committed Jul 9, 2024
2 parents 386273b + 4e2f8c0 commit df75554
Show file tree
Hide file tree
Showing 199 changed files with 7,869 additions and 1,502 deletions.
6 changes: 3 additions & 3 deletions src/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -180,12 +180,12 @@ which installs exactly the right packages, and builds the code.

See `packages/backend/data.ts` . In particular, you can set BASE_PATH, DATA, PGHOST, PGDATA, PROJECTS, SECRETS to override the defaults. Data is stored in `cocalc/src/data/` by default.

#### Filesystem Build Caching
#### File System Build Caching

There are two types of filesystem build caching. These greatly improve the time to compile typescript or start webpack between runs. However, in rare cases bugs may lead to weird broken behavior. Here's where the caches are, so you know how to clear them to check if this is the source of trouble. _As of now, I'm_ _**not**_ _aware of any bugs in filesystem caching._
There are two types of file system build caching. These greatly improve the time to compile typescript or start webpack between runs. However, in rare cases bugs may lead to weird broken behavior. Here's where the caches are, so you know how to clear them to check if this is the source of trouble. _As of now, I'm_ _**not**_ _aware of any bugs in file system caching._

- In the `dist/` subdirectory of a package, there's a file `tsconfig.tsbuildinfo` that caches incremental typescript builds, so running `tsc` is much faster. This is enabled by setting `incremental: true` in `tsconfig.json`. I've never actually seen a case where caching of this file caused a problem (those typescript developers are careful).
- Webpack caches its builds in `/tmp/webpack` . This is configured in `packages/static/webpack.config.js` , and we use `/tmp` since random access file system performance is critical for this **large** GB+ cache -- otherwise, it's almost slower than no cache. (I also benchmarked tsc, and it works fine on a potentially slow local filesystem.) I did sees bugs with this cache when I had some useless antd tree shaking plugin enabled, but I have never seen any problems with it since I got rid of that.
- Webpack caches its builds in `/tmp/webpack` . This is configured in `packages/static/webpack.config.js` , and we use `/tmp` since random access file system performance is critical for this **large** GB+ cache -- otherwise, it's almost slower than no cache. (I also benchmarked tsc, and it works fine on a potentially slow local file system.) I did sees bugs with this cache when I had some useless antd tree shaking plugin enabled, but I have never seen any problems with it since I got rid of that.

#### Creating an admin user

Expand Down
4 changes: 2 additions & 2 deletions src/compute/compute/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,9 @@ pnpm build
- You \(and collabs\) can then use this power from cocalc on your laptop.
- A web browser with WebGPU [providing PyTorch](https://praeclarum.org/2023/05/19/webgpu-torch.html) \(say\).

## The filesystem
## The File System

The filesystem from the project will get mounted via [WebSocketFS](https://github.com/sagemathinc/websocketfs). This will initially only be for FUSE, but later could also use WASI in the browser.
The file system from the project will get mounted via [WebSocketFS](https://github.com/sagemathinc/websocketfs). This will initially only be for FUSE, but later could also use WASI in the browser.

## Status

Expand Down
8 changes: 4 additions & 4 deletions src/compute/compute/lib/filesystem.ts
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
/*
Mount a remote CoCalc project's filesystem locally over a websocket using FUSE.
Mount a remote CoCalc project's file system locally over a websocket using FUSE.
await require('.').mount({remote:'wss://cocalc.com/10f0e544-313c-4efe-8718-2142ac97ad11/raw/.smc/websocketfs',path:process.env.HOME + '/dev2', connectOptions:{perMessageDeflate: false, headers: {Cookie: require('cookie').serialize('api_key', 'sk-at7ALcGBKMbzq7Vc00000P')}}})
Expand Down Expand Up @@ -39,9 +39,9 @@ interface Options {
unionfs?: {
upper: string;
lower: string;
// If true, doesn't do anything until the type of the filesystem that lower is
// If true, doesn't do anything until the type of the file system that lower is
// mounted on is of this type, e.g., "fuse". This is done *INSTEAD OF* just
// trying to mount that filesystem. Why? because in docker we hit a deadlock
// trying to mount that file system. Why? because in docker we hit a deadlock
// when trying to do both in the same process (?), which I can't solve -- maybe
// a bug in node. In any case, separating the unionfs into a separate container
// is nice anyways.
Expand Down Expand Up @@ -160,7 +160,7 @@ export async function mountProject({
// modified = ignore any file modified with this many seconds (at least);
// also ignores any file not in the stat cache.
readTrackingFile: readTrackingFile,
readTrackingExclude: [".*", ...exclude],
readTrackingExclude: exclude,
// metadata file
metadataFile,
}));
Expand Down
6 changes: 3 additions & 3 deletions src/compute/compute/lib/manager.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,11 +40,11 @@ interface Options {
// it should be the id in the database from the compute_servers table.
compute_server_id: number;
// HOME = local home directory. This should be a network mounted (or local)
// filesystem that is identical to the home directory of the target project.
// file system that is identical to the home directory of the target project.
// The ipynb file will be loaded and saved from here, and must exist, and
// process.env.HOME gets set to this.
home: string;
// If true, doesn't do anything until the type of the filesystem that home is
// If true, doesn't do anything until the type of the file system that home is
// mounted on is of this type, e.g., "fuse".
waitHomeFilesystemType?: string;
}
Expand Down Expand Up @@ -117,7 +117,7 @@ class Manager {
this.state = "init";
// Ping to start the project and ensure there is a hub connection to it.
await pingProjectUntilSuccess(this.project_id);
// wait for home direcotry filesystem to be mounted:
// wait for home direcotry file system to be mounted:
if (this.waitHomeFilesystemType) {
this.reportComponentState({
state: "waiting",
Expand Down
2 changes: 1 addition & 1 deletion src/compute/compute/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,6 @@
"devDependencies": {
"@types/cookie": "^0.5.1",
"@types/node": "^18.16.14",
"typescript": "^5.2.2"
"typescript": "^5.5.3"
}
}
Loading

0 comments on commit df75554

Please sign in to comment.