init by baileydunning · Pull Request #1 · HarperFast/image-optimizer

baileydunning · 2025-09-03T22:40:28Z

This api is responsible for uploading and retrieving optimized images using HarperDB as the backend. Key features include:

Upload endpoint (POST /images) for storing original images with format validation.
Retrieval endpoint (GET /images) supporting dynamic resizing, format conversion (WebP, AVIF, JPEG), and device pixel ratio.
Add update endpoint (PUT /images) for upserting original images and purging stale variants
Integrate HarperDB tables (images, image_variants) for storage and caching, as defined in schema.graphql
Configuration via config.yaml
Main API logic in resources.ts

kriszyp

Lots of awesome stuff in here, great work!

kriszyp · 2025-09-03T22:44:47Z

+
+        if (Buffer.isBuffer(target.data)) {
+            bytes = target.data;
+        } else if (typeof target.data?.arrayBuffer === "function") {


This is looking for a Blob, I presume? Do we have any way that this can currently take place, through HTTP?

kriszyp · 2025-09-03T22:50:00Z

+const ImagesTable = databases.ImageOptimization.images;
+const VariantsTable = databases.ImageOptimization.image_variants;
+
+export class Images extends Resource {


Based on the signatures of the methods you are using, I think you want this class to have static loadAsInstance = false.

kriszyp · 2025-09-03T22:50:27Z

+    }
+
+    // Upload and store original image 
+    async post(target: any, data: any) {


It seems like it would be pretty intuitive to have a put() method here too, so the user can actually specify the id/path.

And I think these arguments are getting reversed due to the lack static loadAsInstance = false.

kriszyp · 2025-09-03T22:52:54Z

+        await ImagesTable.put({
+            id,
+            blob: bytes,
+            contentType: target.headers?.["content-type"] || "application/octet-stream",


Also, the data (the real one, which is confusing because the arguments are reversed), has a direct contentType property. But this is fine too.

kriszyp · 2025-09-03T22:56:25Z

+            const variantKey = `${id}_${width || 'orig'}_${dpr}_${format}`;
+
+            // Try to get variant from database
+            let variant = await VariantsTable.get(variantKey);


This should be implemented as a caching table (see https://docs.harperdb.io/docs/developers/applications/caching for docs on this). Manually doing a get and put introduces a lot of race conditions.

Sweet, thank you for the tip

kriszyp · 2025-09-04T11:35:17Z

+            await sharp(bytes).metadata();
+        } catch (err: any) {
+            logger.error("Invalid image format:", err);
+            return { status: 400, data: { error: "Invalid image format: " + err.message } };


A Response object has to have a status and a headers otherwise, it is interpreted as a plain object to serialize. Errors should be thrown, not returned.

If I throw errors in the response, I get 500 status codes for everything. Is there a workaround for this, or a recommended way to implement error handling properly with Harper so client errors (like 400) are returned with the correct status code?

Yes: https://docs.harperdb.io/docs/reference/resources/instance-binding#throwing-errors

kriszyp · 2025-09-04T11:36:42Z

@@ -0,0 +1 @@
+declare module 'harperdb';


Is this necessary for the types? It is not imported, right?

yes I was getting errors until I added this bit in. Stolen from the early hints template

kriszyp · 2025-09-04T11:39:15Z

+
+export class Images extends Resource {
+    allowRead(user: User) {
+        return user?.role?.id === 'super_user';


Wouldn't we typically want to allow broader access to images?

Good point—this was originally pulled from the early hints template. I think it makes the most sense to restrict access for POST and PUT operations, while allowing broader read access for all image variants. It really depends on the usage of the api though, like if it used for something that allows public uploads we'd want to allow wider access to those endpoints as well. For the sake of this template, do you think we should stick with the secure defaults, or make it more permissive to cover broader use cases?

kriszyp · 2025-09-04T11:41:36Z

+            return { status: 400, data: { error: "Invalid image format: " + err.message } };
+        }
+
+        const id = Math.random().toString(36).slice(2, 10);


Harper itself can generate the ids with the create method, which uses UUIDs for strings (provides better guarantees of uniqueness), or auto-incrementation for numbers (which is faster).

That's cool! Are there reference docs for this / other methods somewhere or can I just import it and call create(id). Im trying to find something about it but don't think I'm searching in the right place

I think https://docs.harperdb.io/docs/reference/resources/instance-binding#createrecord-object-context-resourcecontext-promiseresource

… for cache key parsing and content type formatting

… error status properties and adding checks for missing IDs

…dling

Ethan-Arrowood

Excellent work on this!

Ethan-Arrowood · 2025-09-09T22:10:15Z

+      - name: Create ImageOptimization database
+        run: |
+          curl -u admin:password \
+            -H "Content-Type: application/json" \
+            -d '{"operation":"create_database","database":"ImageOptimization"}' \
+            http://localhost:9925
+
+      - name: Create images table
+        run: |
+          curl -u admin:password \
+            -H "Content-Type: application/json" \
+            -d '{"operation":"create_table","database":"ImageOptimization","table":"images","primary_key":"id"}' \
+            http://localhost:9925
+
+      - name: Create image_variants table
+        run: |
+          curl -u admin:password \
+            -H "Content-Type: application/json" \
+            -d '{"operation":"create_table","database":"ImageOptimization","table":"image_variants","primary_key":"id"}' \
+            http://localhost:9925


Are these not being created from the graphqlSchema ?

No I don't think so, it was unable to find the db/tables without this step

Seems to work for me 🤔

Maybe you need to restart after deploying the app.

I tried adding a restart step after deploying but am still seeing connectivity issues without those steps to create the db and tables: failed gh action run

Actually was able to remove them! I realized I was referencing the wrong tables in the resources file instead of the ones created by the schema so once I switched that was able to remove those steps

…eanup step

… schema

Co-authored-by: Ethan Arrowood <ethan@arrowood.dev>

…nd tables

…ource methods for cleaner code

Ethan-Arrowood

nothing to block on. this is awesome. great work

Ethan-Arrowood · 2025-09-17T19:15:10Z

+	if (res.status !== 200) {
+		console.log('Variant error response:', res.status, await res.text());
+	}


Similar to above; I'd rather not mix debug code with test code. Just fail with the assertion

Ethan-Arrowood · 2025-09-17T19:19:31Z

+	const format = formatRaw?.toLowerCase() as VariantFormat;
+
+	if (!imageId || !/^[a-z0-9_-]+$/i.test(imageId)) return null;
+	if (!['webp', 'jpeg', 'avif', 'png'].includes(format)) return null;


You could still use a type assertion function here; just catch the error and return null

try { assertFormat(format); } catch (err) { return null; } // Now `format` is typed as `VariantFormat` here

Co-authored-by: Ethan Arrowood <ethan@arrowood.dev>

…key parsing

Co-authored-by: Ethan Arrowood <ethan@arrowood.dev>

kriszyp

I made a couple of minor notes, just FYI, but these are really minor. Great job with all of this! This is an excellent component!

kriszyp · 2025-09-19T11:41:35Z

+		try {
+			cached = await VariantsTable.get(cacheKey);
+		} catch (err) {
+			logger.error('VariantsTable.get failed:', err);


FYI, Harper should log thrown errors for you, this will likely result in two log entries.

kriszyp · 2025-09-19T11:43:55Z

+			throw err;
+		}
+		if (!image?.blob) {
+			logger.error('Image not found for imageId:', imageId);


This may be a little excessive log level since typically 404s are common and easily triggered by users. Also, you can return undefined to signal the entry is not found and Harper will return a 404 (quicker path since it doesn't involve throwing an error), or a return an object with status/header/body. But this will certainly work.

init

b76b688

baileydunning self-assigned this Sep 3, 2025

kriszyp requested changes Sep 3, 2025

View reviewed changes

fix line

8080284

kriszyp reviewed Sep 4, 2025

View reviewed changes

baileydunning added 25 commits September 4, 2025 09:11

Implement image variant generation and caching; add utility functions…

a52b41c

… for cache key parsing and content type formatting

docs

394e7f1

prettier

b20ef59

fix

c73c2a9

Update allowRead method in ImageVariant to allow access for all users

50c3854

Refactor ImageVariant and Images classes for improved error handling

7e34177

Improve error handling in ImageVariant and Images classes by updating…

a382640

… error status properties and adding checks for missing IDs

Create test and add GitHub Actions workflow for running tests

e7b573f

declare global type to try to make tests happy

cd9be8d

adjust lib in tsconfig

8410fa4

nvm

425df4d

add step to test workflow to install harper

9c36577

declare type

aee5fd7

rm import

0578a96

try globalthis

ce95cfd

add harper as a package

ae38526

nvm

17702c4

go back to this

9af2680

maybe

278f7f2

maybe as a dev dependency

4c21957

rm again

b1cbe75

add steps to install and setup harper

3a451d8

comment out

324a717

organize HarperDB setup steps in CI workflow

dc9dffb

Add integration tests for Image Optimizer API and update resource han…

35556c2

…dling

baileydunning added 6 commits September 8, 2025 19:38

add steps to create db and tables

4091ebc

move deploy step lower

75169d1

enhance integration tests for image variant caching and error handling

3a624e5

update license

d071ee9

add a step to clean up the db

ac9a0ec

update Node.js version retrieval and remove .nvmrc file

7888211

Ethan-Arrowood reviewed Sep 9, 2025

View reviewed changes

baileydunning and others added 13 commits September 10, 2025 09:22

refactor: update file handling in tests and remove unused database cl…

1fa75f2

…eanup step

refactor: improve caching response structure and enhance ImageVariant…

8f04483

… schema

Update README.md

427854b

Co-authored-by: Ethan Arrowood <ethan@arrowood.dev>

Update README.md

081f15b

Co-authored-by: Ethan Arrowood <ethan@arrowood.dev>

Update README.md

03ebec4

Co-authored-by: Ethan Arrowood <ethan@arrowood.dev>

Update README.md

d65b5fe

Co-authored-by: Ethan Arrowood <ethan@arrowood.dev>

Update README.md

4f1c1bb

Co-authored-by: Ethan Arrowood <ethan@arrowood.dev>

Update api/utils/index.ts

e2536d0

Co-authored-by: Ethan Arrowood <ethan@arrowood.dev>

add restart step

5dbbff3

Update GitHub Actions workflow to create ImageOptimization database a…

ca883ba

…nd tables

get cache hits to work

3bd5ad4

Refactor image variant retrieval and error handling in API

5f6c587

Remove database and table creation steps from workflow and update res…

40fcfa9

…ource methods for cleaner code

Ethan-Arrowood approved these changes Sep 17, 2025

View reviewed changes

baileydunning and others added 3 commits September 17, 2025 15:28

Update api/test/resources.test.js

de31e01

Co-authored-by: Ethan Arrowood <ethan@arrowood.dev>

Refactor image format validation and improve error handling in cache …

88ae915

…key parsing

spacing

8a45f87

Ethan-Arrowood approved these changes Sep 18, 2025

View reviewed changes

baileydunning and others added 3 commits September 18, 2025 09:12

Update api/resources.ts

74c0f1a

Co-authored-by: Ethan Arrowood <ethan@arrowood.dev>

Update package.json

56ccfe0

Co-authored-by: Ethan Arrowood <ethan@arrowood.dev>

format

20bbe19

baileydunning merged commit ed502eb into main Sep 18, 2025
1 check passed

kriszyp reviewed Sep 19, 2025

View reviewed changes

		@@ -0,0 +1 @@
		declare module 'harperdb'; No newline at end of file

Conversation

baileydunning commented Sep 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kriszyp left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ethan-Arrowood left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Ethan-Arrowood left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kriszyp left a comment

Choose a reason for hiding this comment

baileydunning commented Sep 3, 2025 •

edited

Loading