Format schema with oneOf doesn't work correctly

[avkos]

const schema = {
  type: 'object',
  properties: {
    from: {
      format: 'address',
    },
    to: {
      oneOf: [{ format: 'string' }, { type: 'null' }],
    },
  },
};

const data = {
  from: '0x7ed0e85b8e1e925600b4373e6d108f34ab38a401',
  to: 123,
};

const res = format(schema, data);

// res { from: '0x7ed0e85b8e1e925600b4373e6d108f34ab38a401', to: 123 }

Actual Behaviour:
to is not formatted

Expected Behaviour:
to should be formatted to a string

[Franklin-Osede]

Hello, I would like to contribute to resolve this issue. Thanks

[mmyyrroonn]

@Franklin-Osede Hi, are you working on this issue? I want to take over it if you're not yet. 😄

[mconnelly8]

Hey @mmyyrroonn, it doesn't look like @Franklin-Osede is working on this. Please go ahead and work on this one. We appreciate you wanting to help out.

[mmyyrroonn]

@avkos @mconnelly8 Hi, after debugging the code, I found the root cause.

The schemaProp would be {"oneOf":[{"format":"string"},{"type":"null"}]} during parsing the to property, which does not have format property. The covertScalarValue would throw an error and skip this property. The code path for this situation is missing in the current codebase.

web3.js/packages/web3-utils/src/formatter.ts

Line 325 in 89711ab

object[key] = convertScalarValue(value, schemaProp.format as string, format);

Here are three ways to fix this bug in my mind.

• fix it in the convertScalarValue function
• fix it in the findSchemaByDataPath function
• fix it in the convert function by adding extra logic, targeting this bug

I would like to know more about this bug and its background before I start my PR. Is there any guideline we should follow for the oneOf keyword? For example, if the schema is to: {oneOf: [{ format: 'string' }, { format: 'number' }],},, what should we do?

[luu-alex]

Hey there @mmyyrroonn thanks for taking a look into this :) to: {oneOf: [{ format: 'string' }, { format: 'number' }],} I haven't dived into the formatter too much, but i dont believe this should be possible, maybe im not understanding why it would have these two different types of format?

[mmyyrroonn]

Hey there @mmyyrroonn thanks for taking a look into this :) to: {oneOf: [{ format: 'string' }, { format: 'number' }],} I haven't dived into the formatter too much, but i dont believe this should be possible, maybe im not understanding why it would have these two different types of format?

Hi, I'm glad to hear your response. I'm not familiar with the schema. what is the definition of the keyword oneOf? The above example may be a wrong or invalid schema according to my personal understanding of the word oneOf. It's much better if there is a doc I can check to follow. Right now, I understand why the situation happened this way. However, I’m unsure about the correct code logic for this keyword. I don't want to fix a bug by creating a new one 😄

[luu-alex]

You bring up a really good point, docs would definitely be helpful in this and its something we probably want in the future :)
In this example

to: {
			oneOf: [{ format: 'address' }, { type: 'null' }],
		},

oneOf allows the formatter match multiple types.
if to is defined, it must be formatted to an address
type describes it that it can also be null.

In this example,

transactions: {
			oneOf: [
				{
					type: 'array',
					items: {
						...transactionInfoSchema,
					},
				},
				{
					type: 'array',
					items: {
						format: 'bytes32',
					},
				},
			],
		},

oneOf allows a schema to be matched by either an array with objects of transactionInfoSchema and will be formatted what is being defined by transactionInfoSchema OR an array of items and must be formatted as bytes32.

Honestly you are being a really big help because this has been an issue thats persisted for quite a while 😅 and being able to identify that this is occuring because format isnt being defined is great.

I think targetting
findSchemaByDataPath or the convert would make most sense to me

[mmyyrroonn]

@luu-alex

Thank you for the examples you provided. After reading them, I would like to confirm a few points:

1. Are the schemas within oneOf mutually exclusive? In other words, if there is a value, it should only match one of the schemas. Is this a precondition I can rely on when writing code?
2. If they are not mutually exclusive, what are the matching principles? For example, does it match the first one that appears? This is why I proposed to: {oneOf: [{ format: 'string' }, { format: 'number' }] }, although after looking at schema.ts, it seems unlikely in practical cases.
3. Will all scenarios be covered in the examples from schema.ts? Do we need to handle more general cases?

[luu-alex]

@mmyyrroonn Yes it should only match only one, and I believe that part is written already so you wouldn't need to worry. I think in this case

to: {
      oneOf: [{ format: 'string' }, { type: 'null' }],
    },

if i understand the error, covertScalarValue will fail because the object {type: 'null'} does not have format. In this scenario if before going into covertScalarValue check type == "null" and isNullish(value) === true don't go into convertScalarValue and just have that property undefined.
Id have to look at this more but I think that makes most sense to me, let me know what you think

[mmyyrroonn]

@luu-alex Let me share more details here for our discussion.
Because 'to' is already a top-level property, there is no recursive call above 'to'. Therefore, before executing the next line of code, dataPath is exactly 'to'.

web3.js/packages/web3-utils/src/formatter.ts

Lines 290 to 293 in 5ad7e5b

	for (const [key, value] of Object.entries(object)) {
	dataPath.push(key);
	const schemaProp = findSchemaByDataPath(schema, dataPath, oneOfPath);

The output of findSchemaByDataPath is {"oneOf":[{"format":"string"},{"type":"null"}]} instead of {"format":"string"}, which, in my view, is incorrect. I am unsure if this result aligns with the original intent of the function or if oneOf should be handled correctly here. I will explain the specific logic of this function below.

web3.js/packages/web3-utils/src/formatter.ts

Lines 295 to 311 in 5ad7e5b

	if (isNullish(schemaProp)) {
	delete object[key];
	dataPath.pop();
	continue;
	}
	// If value is an object, recurse into it
	if (isObject(value)) {
	convert(value, schema, dataPath, format, oneOfPath);
	dataPath.pop();
	continue;
	}
	// If value is an array
	if (
	convertArray({

Since schemaProp is not null and the value is 123, which is not object or array, the above logic would be skipped and go into the convertScalarValue function.

web3.js/packages/web3-utils/src/formatter.ts

Line 325 in 5ad7e5b

object[key] = convertScalarValue(value, schemaProp.format as string, format);

The schemaProp.format is undefined and the convertScalarValue function would go into the error catch path.

web3.js/packages/web3-utils/src/formatter.ts

Line 134 in 5ad7e5b

} catch (error) {

Returning to this function, dataPath consists of only one item, so the loop will iterate only once and not twice. In this iteration, previousDataPath is empty, so the logic related to oneOf at line 59 is skipped, and lines 71 and 83 are executed directly. After the loop ends, the function returns. At this point, the result is {"oneOf":[{"format":"string"},{"type":"null"}]}.

web3.js/packages/web3-utils/src/formatter.ts

Lines 50 to 86 in 5ad7e5b

	const findSchemaByDataPath = (
	schema: JsonSchema,
	dataPath: string[],
	oneOfPath: [string, number][] = [],
	): JsonSchema | undefined => {
	let result: JsonSchema = { ...schema } as JsonSchema;
	let previousDataPath: string | undefined;
	for (const dataPart of dataPath) {
	if (result.oneOf && previousDataPath) {
	const currentDataPath = previousDataPath;
	const path = oneOfPath.find(([key]) => key === currentDataPath);
	if (path && path[0] === previousDataPath) {
	// eslint-disable-next-line @typescript-eslint/no-unsafe-assignment, @typescript-eslint/no-unsafe-member-access
	result = result.oneOf[path[1]];
	}
	}
	if (!result.properties && !result.items) {
	return undefined;
	}
	if (result.properties) {
	result = (result.properties as Record<string, JsonSchema>)[dataPart];
	} else if (result.items && (result.items as JsonSchema).properties) {
	const node = (result.items as JsonSchema).properties as Record<string, JsonSchema>;
	result = node[dataPart];
	} else if (result.items && isObject(result.items)) {
	result = result.items;
	} else if (result.items && Array.isArray(result.items)) {
	result = result.items[parseInt(dataPart, 10)];
	}
	if (result && dataPart) previousDataPath = dataPart;
	}
	return result;

I'm unsure if the findSchemaByDataPath function should handle this scenario because in this case, the oneOfPath is also empty.
In my view, the handling logic for this example has been overlooked. Perhaps we should modify this logic to include additional handling.

web3.js/packages/web3-utils/src/formatter.ts

Lines 303 to 306 in 5ad7e5b

	if (isObject(value)) {
	convert(value, schema, dataPath, format, oneOfPath);
	dataPath.pop();
	continue;

let me know what you think 😄

[luu-alex]

Ah I understand it a bit more, sure It makes sense to include this additional handling. Feel free to start on the PR

[mmyyrroonn]

Ah I understand it a bit more, sure It makes sense to include this additional handling. Feel free to start on the PR

working on it

[mmyyrroonn]

@luu-alex Hi, I created a PR to demonstrate the fix. Actually, there are two issues: one is the problem I described above. The other is that string is not processed in the original logic; I’m not sure if this is an issue or if the issue itself is not actually a problem.

[luu-alex]

@mmyyrroonn Thanks for the PR! Ill take a look