Support for fixedscaleoffset

[abarciauskas-bgse]

👋🏽 wondering how hard it would be to add support for fixedscaleoffset? I am playing with zarrita.js and the POWER Zarr dataset on AWS (for example https://nasa-power.s3.us-west-2.amazonaws.com/merra2/spatial/power_merra2_annual_spatial_utc.zarr, variable TS) and zarrita.js is unable to load the dataset because fixedscaleoffset is an unknown codec.

[manzt]

Hi there, the implementation looks fairly straight forward (https://github.com/zarr-developers/numcodecs/blob/main/numcodecs/fixedscaleoffset.py). Since it's not an especially complicated codec, I'd be happy to start with having an implementation builtin to zarrita.js (just an decode method as an bytes_to_array
codec).

We have several codecs that are simple and only implement the decode method:
https://github.com/manzt/zarrita.js/blob/main/packages/core/src/codecs/vlen-utf8.ts

[manzt]

import * as zarr from "jsr:@zarrita/zarrita@0.4.0-next.19";
import { assert } from "jsr:@std/assert@1.0.11";

function parseSupportedDataType(dtype: string) {
	const match = dtype.match(/^([<|>])(.*)$/);
	assert(match, `Invalid dtype: ${dtype}`);
	const [, endian, rest] = match;
	const dataType = ({
		i1: "Int8",
		u1: "Uint8",
		i2: "Int16",
		u2: "Uint16",
		i4: "Int32",
		u4: "Uint32",
		// TODO: requires special handling in the decoder between bigint/number
		// i8: "BigInt64",
		// u8: "BigUint64",
		f4: "Float32",
		f8: "Float64",
	} as const)[rest];
	assert(dataType, `Unsupported data type: ${dtype}`);
	assert(endian !== ">", "bigint not yet supported.");
	return dataType;
}

type FixedScaleOffsetCodecConfig = {
	/** Value to subtract from data */
	offset: number;
	/** Value to multiply by data.*/
	scale: number;
	/** Data type to use for decoded data. */
	dtype: string;
	/** Data type to use for encoded data. */
	astype: string;
};

export class FixedScaleOffsetCodec {
	readonly kind = "bytes_to_bytes";
	#offset: number;
	#scale: number;
	#dtype: ReturnType<typeof parseSupportedDataType>;
	#astype: ReturnType<typeof parseSupportedDataType>;

	constructor(configuration: FixedScaleOffsetCodecConfig) {
		this.#scale = configuration.scale;
		this.#offset = configuration.offset;
		this.#dtype = parseDataType(configuration.dtype);
		this.#astype = parseDataType(configuration.dtype);
	}

	static fromConfig(configuration: FixedScaleOffsetCodecConfig) {
		return new FixedScaleOffsetCodec(configuration);
	}

	encode(_chunk: zarr.Chunk<D>): Uint8Array {
		throw new Error("Method not implemented.");
	}

	decode(bytes: Uint8Array): Uint8Array {
		let input = new globalThis[`${this.#astype}Array`](
			bytes.buffer as ArrayBuffer,
		);
		let output = new globalThis[`${this.#dtype}Array`](input.length);
		input.forEach((value, i) => {
			output[i] = (value / this.#scale) + this.#offset;
		});
		return new Uint8Array(input.buffer);
	}
}

I haven't tested but something like this... You could add this to your own program if interested to see if it works:

import * as zarr from "jsr:@zarrita/zarrita@0.4.0-next.19";
import { FixedScaleOffsetCodec } from "./fixedscaleoffset.ts";

zarr.registry.set("fixedscaleoffset", () => FixedScaleOffsetCodec);

zarr.open(...)

, and then we could add upstream in zarrita..

[abarciauskas-bgse]

Thank you @manzt ! I would love to look into this more but afraid it's not of my current priorities. We are going to work with zarrita.js (my colleagues Hanbyul and Max Jones specifically) but will likely not use the POWER dataset.

[abarciauskas-bgse]

Should I close this issue for now?