Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add function isValidJSON #7293

Merged
merged 6 commits into from
Oct 21, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
1 change: 1 addition & 0 deletions dbms/src/Functions/FunctionsJSON.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ namespace DB
void registerFunctionsJSON(FunctionFactory & factory)
{
factory.registerFunction<FunctionJSON<NameJSONHas, JSONHasImpl>>();
factory.registerFunction<FunctionJSON<NameIsValidJSON, IsValidJSONImpl>>();
factory.registerFunction<FunctionJSON<NameJSONLength, JSONLengthImpl>>();
factory.registerFunction<FunctionJSON<NameJSONKey, JSONKeyImpl>>();
factory.registerFunction<FunctionJSON<NameJSONType, JSONTypeImpl>>();
Expand Down
31 changes: 31 additions & 0 deletions dbms/src/Functions/FunctionsJSON.h
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,7 @@ class FunctionJSON : public IFunction


struct NameJSONHas { static constexpr auto name{"JSONHas"}; };
struct NameIsValidJSON { static constexpr auto name{"isValidJSON"}; };
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the name JSONIsValid() would be more consistent with other names.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. BTW, isValidJSON is consistent with isValidUTF8.

struct NameJSONLength { static constexpr auto name{"JSONLength"}; };
struct NameJSONKey { static constexpr auto name{"JSONKey"}; };
struct NameJSONType { static constexpr auto name{"JSONType"}; };
Expand Down Expand Up @@ -311,6 +312,36 @@ class JSONHasImpl
};


template <typename JSONParser>
class IsValidJSONImpl
{
public:
static DataTypePtr getType(const char * function_name, const ColumnsWithTypeAndName & arguments)
{
if (arguments.size() != 1)
{
/// IsValidJSON() shouldn't get parameters other than JSON.
throw Exception{"Function " + String(function_name) + " needs exactly one argument",
ErrorCodes::NUMBER_OF_ARGUMENTS_DOESNT_MATCH};
}
return std::make_shared<DataTypeUInt8>();
}

using Iterator = typename JSONParser::Iterator;
static bool addValueToColumn(IColumn & dest, const Iterator &)
{
/// This function is called only if JSON is valid.
/// If JSON isn't valid then `FunctionJSON::Executor::run()` adds default value (=zero) to `dest` without calling this function.
ColumnVector<UInt8> & col_vec = assert_cast<ColumnVector<UInt8> &>(dest);
col_vec.insertValue(1);
return true;
}

static constexpr size_t num_extra_arguments = 0;
static void prepare(const char *, const Block &, const ColumnNumbers &, size_t) {}
};


template <typename JSONParser>
class JSONLengthImpl
{
Expand Down
4 changes: 4 additions & 0 deletions dbms/tests/queries/0_stateless/00918_json_functions.reference
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,10 @@
1
1
0
--isValidJSON--
1
0
0
--JSONKey--
a
b
Expand Down
5 changes: 5 additions & 0 deletions dbms/tests/queries/0_stateless/00918_json_functions.sql
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,11 @@ SELECT JSONHas('{"a": "hello", "b": [-100, 200.0, 300]}', 'a');
SELECT JSONHas('{"a": "hello", "b": [-100, 200.0, 300]}', 'b');
SELECT JSONHas('{"a": "hello", "b": [-100, 200.0, 300]}', 'c');

SELECT '--isValidJSON--';
SELECT isValidJSON('{"a": "hello", "b": [-100, 200.0, 300]}');
SELECT isValidJSON('not a json');
SELECT isValidJSON('"HX-=');
Copy link
Member

@vitlibar vitlibar Oct 14, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like isValidJSON('json') works exactly as JSONHas('json'). So maybe it would be the better solution not to add another function, but add an alias

factory.registerAlias("isValidJSON", "JSONHas");

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then you'll be able to pass additional argument to isValidJSON. I don't know if it's appropriate.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems it's all about the name:
the JSONIsValid name can be considered as the function checking not only a passed json, but passed indices too (JSONIsValid('json' [, indices]));
the IsValidJSON() name looks more like it's just a check for a passed json only (IsValidJSON('json').
But we already have the function JSONHas(), so let's keep adding IsValidJSON() with a single 'json' argument in this PR.


SELECT '--JSONKey--';
SELECT JSONKey('{"a": "hello", "b": [-100, 200.0, 300]}', 1);
SELECT JSONKey('{"a": "hello", "b": [-100, 200.0, 300]}', 2);
Expand Down
11 changes: 11 additions & 0 deletions docs/en/query_language/functions/json_functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,17 @@ There is currently no support for code points in the format `\uXXXX\uYYYY` that

The following functions are based on [simdjson](https://github.com/lemire/simdjson) designed for more complex JSON parsing requirements. The assumption 2 mentioned above still applies.

## isValidJSON(json)

Checks that passed string is a valid json.

Examples:

```sql
SELECT isValidJSON('{"a": "hello", "b": [-100, 200.0, 300]}') = 1
SELECT isValidJSON('not a json') = 0
```

## JSONHas(json[, indices_or_keys]...)

If the value exists in the JSON document, `1` will be returned.
Expand Down
11 changes: 11 additions & 0 deletions docs/ru/query_language/functions/json_functions.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,17 @@ visitParamExtractString('{"abc":"hello}', 'abc') = ''

Следующие функции используют [simdjson](https://github.com/lemire/simdjson) который разработан по более сложны требования для разбора JSON. Упомянутое выше предположение 2 по-прежнему применимо.

## isValidJSON(json)

Проверяет, является ли переданная строка валидным json значением.

Примеры:

```sql
SELECT isValidJSON('{"a": "hello", "b": [-100, 200.0, 300]}') = 1
SELECT isValidJSON('not a json') = 0
```

## JSONHas(json[, indices_or_keys]...)

Если значение существует в документе JSON, то возвращается `1`.
Expand Down