Is converting values in the scope of this library ?

[antoinewdg]

A common use case when validating data from external APIs is to have dates encoded as strings.
Let's say I get the following data from a JSON API:

{ "date": "2015-10-12 12:00:00" }

I would want to represent it with the following Runtype:

Record({ date: InstanceOf(Date) })

Is there a simple way to go from the first to the second without having to use an intermediate Record({ date: String }) runtype, and doing the validation ?

I think that having a convertFrom method on the Runtype interface would help a lot.

interface Runtype<A> {
  // ...
  convertFrom<B> : (checkBefore: (x: any) => B) => Runtype<A>;
}

The returned Runtype would be exactly the same, but the check method would be changed to include the additional validation in checkBefore before the one from the initial Runtype.

I could then do:

const Input = Record({ 
  date: InstanceOf(Date).convertFrom(x => {
    const strX = String.check(x);
    const timestamp = Date.parse(strX);
    if(isNan(timestamp)) {
      throw new ValidationError('Invalid date string ' + strX);
    }
    return new Date(timestamp);
 })
});

Do you think it could be added to the library (I would be willing to do it)?

[pelotom]

It would be a departure from the invariant so far that runtypes never produce new values, they just validate existing ones. That said, I can't think of any specific reason that it's a bad idea.

However before you invest time in implementing it, I feel honor bound to mention that io-ts already has this feature, so it might be a better fit for your use case 🙂

[antoinewdg]

It would be a departure from the invariant so far that runtypes never produce new values, they just validate existing ones. That said, I can't think of any specific reason that it's a bad idea.

I completely agree, it would feel a little hacky given that this is not how it works everywhere else in the lib.

However before you invest time in implementing it, I feel honor bound to mention that io-ts already has this feature, so it might be a better fit for your use case

Thanks for mentioning it! I actually came here looking for an alternative to io-ts, and found I liked runtypes better (which is only my subjective opinion, io-ts also seems great).

Anyway, thanks for the awesome lib! :)

[danielo515]

I usually rely on validation libraries to attempt some conversions (Joi for example does this), and return a possible errors and the converted value.
However, if the target of this library is to only check if a type is valid or not, I can totally understand your position. However, I had to open the issues and search for this particular one, if you are already positioned about this I think it will be nice to include it on the readme.

Regards

[brandonkal]

This would be interesting for string => number conversion and 'false' => false conversion.

[MicahZoltu]

I gave io-ts a try, but its interface is much more verbose and (IMO) harder to read than runtypes. This is the one major feature that would make runtypes totally awesome and give me no desire to look for any other options. Ideal interface would look something like a withConverter function:

const Apple = Record({
	color: Union(Literal('red'), Literal('green'), Literal('yellow'), // 'red' | 'green' | 'yellow'
	seeds: String.withConverter(x => Number.parseFloat(x)), // number
})

This way when I'm working with an API that gives me numbers as strings, or byte arrays as hex strings, or any other non-native JSON serialization I can just throw a withConverter on it. Presumably if withConverter throws, that would be caught by runtypes and wrapped in a helpful error that tells me what property failed to convert.

Would it be possible to build this as an another library so this library doesn't have to get into the business of decoding?

[yuhr]

I gave io-ts a try, but its interface is much more verbose and (IMO) harder to read than runtypes. This is the one major feature that would make runtypes totally awesome and give me no desire to look for any other options.

Totally agree 🤣

I'm very interested in this feature and would love to start implementing.

[MicahZoltu]

@yuhr funtypes (a fork of runtypes) has this feature.

[yuhr]

Got it, here's that. It's excellent effort, but deprecating check in favor of parse makes me feel a bit pointless, and I don't understand why ParsedValue is taking duplicate logic such as parse and test, rather than taking just parse and leaving test to Constraint? Could someone explain these points?

[yuhr]

Let's compare, funtypes looks like this:

const TrimmedString = ParsedValue(String, {
  name: 'TrimmedString',
  parse(value) {
    return { success: true, value: value.trim() };
  },
  test: String.withConstraint(
    value =>
      value.trim() === value || `Expected the string to be trimmed, but this one has whitespace`,
  ),
});

TrimmedString.safeParse(' foo bar ')
// => 'foo bar'

With #191, the equivalent code in runtypes will look like this:

const TrimmedString = String.withTransform(value => value.trim(), { name: 'TrimmedString' });
TrimmedString.check(' foo bar ');
// => 'foo bar'

const AlreadyTrimmedString = String.withConstraint(
  value =>
    value.trim() === value || `Expected the string to be trimmed, but this one has whitespace`,
);

As illustrated above, I think the feature like test in funtypes (it guards the input is already conforming to TrimmedString) should be considered as a different matter than transformation. The same applies to inverse transformation (i.e. serialize in funtypes), obviously it can sufficiently achieved by having another runtype dedicated to perform inverse transformation.

[MicahZoltu]

You don't have to use a custom parser. In your example, it looks like you don't want a custom parser, you just want a constraint.

const TrimmedString = String.withConstraint(x => x.trim() === value)
TrimmedString.parse(' foo bar ') // throws error
TrimmedString.safeParse(' foo bar ') // returns error (IIRC, I never use this one so I forget how it surfaces errors exactly)

[yuhr]

I couldn't get the point of your comment. Apparently your code has no "transform", so it does not work the same way as mine. What I tried to illustrate by my example above is nothing else but the equivalent code corresponding to funtypes' code example. What I meant by the previous post is, I think we should follow the "separation of concerns" principle here.

By the way, the equivalent code to yours in runtypes should look like:

const TrimmedString = String.withConstraint(x => x.trim() === value)
TrimmedString.check(' foo bar ') // throws error
TrimmedString.validate(' foo bar ') // returns error

Or, if I reuse my example code, I can write it also:

AlreadyTrimmedString.check(' foo bar ') // throws error
AlreadyTrimmedString.validate(' foo bar ') // returns error

All these are the concern of "conformity after trimming", not the concern of "transformation of values".

[MicahZoltu]

We should probably move this conversation over to funtypes repository. Feel free to @mention me if you create a thread over there! (I believe that you may be misunderstanding the rename of check to parse, but I don't want to derail this thread/repository since it is a funtypes thing)

[WilliamABradley]

I had a need for this, since I wanted to use the runtypes for data coming in from our request body, query, etc.

Since runtypes doesn't support type conversions out of the box, and I'm not a fan of uprooting all of the runtypes and switch to funtypes, yup, or introduce io-ts, I created support for type conversion using the reflection information of runtypes.

The only requirement is for a runtype to have a brand, so that it's type converter could be resolved properly, but that could be removed from this code.

Implementation:

/**
 * Explore the Runtypes reflection to find a type brand.
 * @param reflect type info to search for brand string in.
 * @returns brand string if found, otherwise, null.
 */
export function findTypeBrands(reflect: rt.Reflect): string[] | undefined {
  switch (reflect.tag) {
    case 'brand':
      return [reflect.brand];

    case 'optional':
      return findTypeBrands(reflect.underlying);

    case 'union': {
      const brands = reflect.alternatives
        .map((alt) => findTypeBrands(alt.reflect))
        .flat()
        .filter((val) => !!val) as string[];
      return brands.length > 0 ? brands : undefined;
    }

    case 'intersect': {
      const brands = reflect.intersectees
        .map((entry) => findTypeBrands(entry.reflect))
        .flat()
        .filter((val) => !!val) as string[];
      return brands.length > 0 ? brands : undefined;
    }

    case 'constraint':
      return findTypeBrands(reflect.underlying);

    default:
      return undefined;
  }
}

/**
 * Gets the array's element type from reflection, or undefined if not an array.
 * @param reflect array type to confirm that the type is an array.
 * @returns Array's element type or undefined if not an array.
 */
export function getArrayElementType(reflect: rt.Reflect): rt.Reflect | undefined {
  switch (reflect.tag) {
    case 'brand':
      return getArrayElementType(reflect.entity);

    case 'optional':
      return getArrayElementType(reflect.underlying);

    case 'union':
      // Only dig deeper if this is a nullable union.
      if (reflect.alternatives.length === 2 && reflect.alternatives[1] === rt.Null) {
        return getArrayElementType(reflect.alternatives[0].reflect);
      }
      return undefined;

    case 'constraint':
      return getArrayElementType(reflect.underlying);

    case 'array':
      return reflect.element;
  }
  return undefined;
}

/**
 * Gets the tuple's component types from reflection, or undefined if not a tuple.
 * @param reflect tuple type to confirm that the type is a tuple.
 * @returns Tuple's component types or undefined if not a tuple.
 */
export function getTupleComponentTypes(reflect: rt.Reflect): rt.Reflect[] | undefined {
  switch (reflect.tag) {
    case 'brand':
      return getTupleComponentTypes(reflect.entity);

    case 'optional':
      return getTupleComponentTypes(reflect.underlying);

    case 'union':
      // Only dig deeper if this in a nullable union.
      if (reflect.alternatives.length === 2 && reflect.alternatives[1] === rt.Null) {
        return getTupleComponentTypes(reflect.alternatives[0].reflect);
      }
      return undefined;

    case 'constraint':
      return getTupleComponentTypes(reflect.underlying);

    case 'tuple':
      return reflect.components;
  }
  return undefined;
}

/**
 * Gets the record's type from reflection, or undefined if not a record.
 * @param reflect record type to confirm that the type is a record.
 * @returns record type or undefined if not a record.
 */
export function getRecordType(reflect: rt.Reflect): (rt.Reflect & { tag: 'record' }) | undefined {
  switch (reflect.tag) {
    case 'brand':
      return getRecordType(reflect.entity);

    case 'optional':
      return getRecordType(reflect.underlying);

    case 'union':
      // Only dig deeper if this in a nullable union.
      if (reflect.alternatives.length === 2 && reflect.alternatives[1] === rt.Null) {
        return getRecordType(reflect.alternatives[0].reflect);
      }
      return undefined;

    case 'constraint':
      return getRecordType(reflect.underlying);

    case 'record':
      return reflect;
  }
  return undefined;
}

/**
 * Gets the insersectee types from reflection, or undefined if not an intersection.
 * @param reflect intersection type to confirm that the type is an intersection.
 * @returns intersectee types or undefined if not an intersection.
 */
export function getIntersecteeTypes(reflect: rt.Reflect): rt.Reflect[] | undefined {
  switch (reflect.tag) {
    case 'brand':
      return getIntersecteeTypes(reflect.entity);

    case 'optional':
      return getIntersecteeTypes(reflect.underlying);

    case 'union':
      // Only dig deeper if this in a nullable union.
      if (reflect.alternatives.length === 2 && reflect.alternatives[1] === rt.Null) {
        return getIntersecteeTypes(reflect.alternatives[0].reflect);
      }
      return undefined;

    case 'constraint':
      return getIntersecteeTypes(reflect.underlying);

    case 'intersect':
      return reflect.intersectees;
  }
  return undefined;
}

/**
 * Runtype type converter storage.
 */
const typeConverters: Record<
  string,
  {
    type: RuntypeBase<unknown>;
    convert: (data: unknown) => unknown;
  }
> = {};

/**
 * Register a runtype with a conversion function. (Must have a brand associated)
 * @param type Type to register for conversion.
 * @param converter Conversion function from untyped to typed.
 */
export function registerTypeConverter(type: RuntypeBase<unknown>, converter: (data: unknown) => unknown) {
  const brands = findTypeBrands(type.reflect);
  if (!brands) {
    throw new Error('Type must have a brand to be registered as a type converter');
  }

  const foundBrand = Object.keys(typeConverters).find((brand) => brands.includes(brand));
  if (foundBrand) {
    throw new Error(`A Type converter was already registered for this type: ${foundBrand}`);
  }

  for (const brand of brands) {
    typeConverters[brand] = {
      type,
      convert: converter,
    };
  }
}

/**
 * Parses a runtype, allowing for type conversions.
 * @param type RunType to parse.
 * @param data Data to parse to RunType
 * @param options Conversion options: {
 *  stripUnknown: "Removes fields that aren't on the type.",
 *  check: "Apply runtype check? Turn to false if you need a higher check, e.g. parsing deep records."
 * }
 * @returns Parsed RunType
 */
export function parseType<T extends RuntypeBase<unknown>>(
  type: T,
  data: unknown,
  { stripUnknown = true, check = true }: { stripUnknown?: boolean; check?: boolean } = {},
): rt.Static<RuntypeBase<T extends RuntypeBase<infer R> ? R : never>> {
  let value = data;

  // Ignore type warnings, used for chaining. Return value if type information couldn't be found.
  // eslint-disable-next-line @typescript-eslint/no-explicit-any
  if (!type) return value as any;

  const brands = findTypeBrands(type.reflect);
  if (brands) {
    for (const brand of brands) {
      const typeConvertInfo = typeConverters[brand];
      if (typeConvertInfo) {
        value = typeConvertInfo.convert(value);
      }
    }
  } else if (Array.isArray(value)) {
    const arrayElementType = getArrayElementType(type.reflect);
    const tupleTypes = getTupleComponentTypes(type.reflect);

    if (arrayElementType) {
      value = value.map((element) => parseType(arrayElementType, element, { check: false, stripUnknown }));
    } else if (tupleTypes) {
      value = value.map((element, index) => parseType(tupleTypes[index], element, { check: false, stripUnknown }));
    }
  } else if (value !== null && value !== undefined && typeof value === 'object') {
    const recordType = getRecordType(type.reflect);
    const intersecteeTypes = getIntersecteeTypes(type.reflect);

    // Handle intersections of types.
    if (intersecteeTypes) {
      const intersectResults: Record<string, unknown>[] = [];
      for (const intersecteeType of intersecteeTypes) {
        intersectResults.push(
          parseType(intersecteeType, value, { check: false, stripUnknown }) as Record<string, unknown>,
        );
      }
      // eslint-disable-next-line @typescript-eslint/no-explicit-any
      value = (_.merge as any)(...intersectResults);
    } else if (recordType) {
      const data = value as Record<string, unknown>;
      const dataKeys = Object.keys(data);
      const record: Record<string, unknown> = stripUnknown ? {} : data;

      for (const [field, fieldType] of Object.entries(recordType.fields)) {
        if (stripUnknown && !dataKeys.includes(field)) continue;
        record[field] = parseType(fieldType as RuntypeBase<unknown>, data[field], { check: false, stripUnknown });
      }

      value = record;
    }
  }

  // We essentially have done everything, if check is false,
  // then we essentially have "casted" the value.
  // eslint-disable-next-line @typescript-eslint/no-explicit-any
  return (check ? type.check(value) : value) as any;
}

Usage:

export const DateString = rt.String.withBrand('DateString').withConstraint((str) => {
  const dateObj = DateTime.fromISO(str);
  return dateObj.isValid || dateObj.invalidExplanation || `The date was not a valid ISO string: ${str}`;
});
export type DateString = rt.Static<typeof DateString>;

export const DateType = rt.InstanceOf(Date).withBrand('Date');
export type DateType = rt.Static<typeof DateType>;

registerTypeConverter(DateType, (value) => {
  if (typeof value === 'string') {
    value = new Date(DateString.check(value));
  }
  return value;
});

const nullableDateType = DateType.nullable().optional();

describe('parseType()', () => {
  const testType = rt.Record({
    date: nullableDateType,
  });

  it('converts values in a record type', () => {
    const source = {
      date: '2022-01-18T23:03:16.328Z',
    };

    expect(parseType(testType, source)).toEqual({
      date: new Date(source.date),
    });
  });
});

[the-spyke]

I understand that having only a single responsibility is good, but knowing where these type checks are used, transformation is a very often second step. Maybe this could be done by a separate types Codec/Decoder/Encoder that can add encoding/decoding by wrapping provided Runtype.