updating S3 engine #519
Replies: 8 comments 2 replies
-
|
Nice work! I actually have @aws-sdk/client-s3 on my TODO list for a future pixl-server-storage version, but just haven't gotten around to it. Yeah, packaging Cronicle into a small bundle or binary executable is a big project, so I'm tackling it in Orchestra. There are a lot of considerations, especially in how the UI works and pulls in JS/CSS resources, most of which are in node_modules and symlinked (see https://github.com/jhuckaby/Cronicle/blob/master/sample_conf/setup.json#L102-L124 ). I've looked at pkg and caxa, and still deciding on the best solution. They all have tradeoffs. |
Beta Was this translation helpful? Give feedback.
-
|
Yeah, single binary is a great options. Although some node_modules clean up would be useful before you pack it up all together. main.js import { DataSet } from "vis-data/peer";
import { Network } from "vis-network/peer";
import "vis-network/styles/vis-network.css";
let nodes = new DataSet([
{ id: 1, label: "Node 1" },
{ id: 2, label: "Node 2" },
{ id: 3, label: "Node 3" },
{ id: 4, label: "Node 4" },
{ id: 5, label: "Node 5" },
])
// @ts-ignore
let edges = new DataSet([
{ from: 1, to: 3 },
{ from: 1, to: 2 },
{ from: 2, to: 4 },
{ from: 2, to: 5 },
{ from: 3, to: 3 },
])
let container = document.getElementById("mynetwork");
let data = {
nodes: nodes,
edges: edges,
};
let options = {};
// @ts-ignore
let network = new Network(container, data, options)index.html <!DOCTYPE html>
<html lang="en">
<head>
<title>Network</title>
<script type="module" src="_combo.js"></script>
<link rel="stylesheet" type="text/css" href="_combo.css" />
<style type="text/css">
#mynetwork {
width: 600px;
height: 400px;
border: 1px solid lightgray;
}
</style>
</head>
<body>
<div id="mynetwork"></div>
</body>
</html>build with esbuild: esbuild --bundle --minify --outfile=_combo.js main.jsBundler will pack those minified libraries in your combo.js, no need to copy/link them. Then you can drop node_modules (with 60MB trash). |
Beta Was this translation helpful? Give feedback.
-
|
FYI, I just upgraded pixl-server-storage to use Thank you so much for your s3engine project. I used this as the basis for mine, as the official docs suck. 🙏🏻 |
Beta Was this translation helpful? Give feedback.
-
|
Hello,
we had to update our configuration by adding protocol in endpoint field and add "forcePathStyle":
Can you add a note on the release for this breaking change? |
Beta Was this translation helpful? Give feedback.
-
|
I found another issue. I'm trying to fix it by adding in S3.js the stream length for the putStream method: It works when I test with a static value, but I have not found yet how to do it dynamically. Any Idea ? |
Beta Was this translation helpful? Give feedback.
-
Yeah, that's because aws sdk V3 doesn't work the same way as before if you trying to put streams. You need to use @aws-sdk/lib-storage now: |
Beta Was this translation helpful? Give feedback.
-
|
Whoops! Damn... I ran all my unit tests on pixl-server-storage pointed at an S3 backend, which includes a bunch of stream tests, and they all passed. I guess Cronicle does something custom... So yeah, I did know about this caveat of @aws-sdk/client-s3, but I was trying to avoid pulling in another heavy dependency. As it turns out, if you pass it a filesystem stream, it works just fine without the extra lib (it's able to glean the data length from it). But it fails on other types of streams, like gzip streams, where it has no way of knowing the length. This is the case with Cronicle, as it streams the job log to gzip, and then streams that stream to S3. Sigh... It will be fixed shortly. I'll also add a note in the docs about users with non-AWS S3 endpoints. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
See if this does the trick → https://github.com/jhuckaby/Cronicle/releases/tag/v0.9.14 |
Beta Was this translation helpful? Give feedback.
-
|
It works for us. Thanks a lot :) |
Beta Was this translation helpful? Give feedback.
-
Current S3 engine for cronicle storage is using aws-sdk package which is pretty huge, about 150MB, while cronicle itself is about 50BM. It was bugging me for a while, so I finally rewrite this engine using newer modular package @aws-sdk/client-s3 (version3 api), which is just 18MB. However there is even better option - you can bundle s3 client and engine into single bundle using esbuild tool. Minified version is just 800KB (so you can replace S3.js file with this bundle without installing any extra packages).
So whoever is using S3 (AWS, minio, ceph, etc) as a storage go ahead and try and test. Here is a repo with new engine and some demos
https://github.com/mikeTWC1984/s3engine
@jhuckaby btw, I tried to bundle entire cronicle using esbuild and it even creates some 5MB bundle, which actually works (unless you minify it), however it still depends on node_modules (unlike my s3 engine project). Not sure what's the problem, will keep experimenting. This could be interesting option for distributing cronicle/orchestra. I'm quite sure cronicle doesn't need most of the stuff from node_modules, there is a lot of .map files and stuff like that.
Beta Was this translation helpful? Give feedback.
All reactions