Proposed JSON Schema Changes for Discussion

[WebBreacher]

There are several bigger changes to the JSON file I have not moved on recently and I want to put them in one place and discuss and then act.

1. Invalid char in username for some sites Invalid char in username for some sites #673 - Add an optional strip_bad_char parameter with a value of whatever characters cause issues. For instance, if a period (.) causes problems, that line would be "strip_bad_char" : ".",. Whatever characters are in that field would be removed from the username prior to making the request to the destination site.
2. Change the category XXXPORNXXX to xx NSFW xx to reflect that not all NSFW (Not Safe For Work) sites are pornographic in nature.
3. Change valid parameter to optional
4. Remove all valid=True entries. Only time we will use the valid parameter would be when we want to temporarily turn off a site. otherwise checkers will assume that if no valid parameter is present then valid=True.

I'm looking for feedback until Thursday, August 17, 2023. At that point, we will move forward with the changes or what has been decided in this issue.

[swedishmike]

Looks good to me - will just have to change validation of 'valid' a bit and that's not a biggie.

[C3n7ral051nt4g3ncy]

Looks good!
Once the JSON is modified, I will then modify it also for my tool.

[AccentuSoft]

Not sure about point 1. Having an optional bad_chars (or valid_chars) field where required seems like a better solution that a binary choice of restricting users to that set.

While it may not matter at this point, I think the flexibility could be important down the line to handle edge cases, i.e. where sites allow only certain special characters. If I am looking for !_john.smith_! for example, and one of the sites does not allow dots, is it not preferable to search for !_johnsmith_! rather than johnsmith?

Alternatively, if '.' is the only character that causes issues as discussed in #673, the invalid char field could just be about stripping the dot. That way we preserve other special characters. This also allows users to get output that matches their original input as closely as possible; for example, if someone puts in !johnsmith! vs johnsmith. The user can always run the second query if the first does not give any results, but if we always strip all special characters, then the user will never get any results that only match the first query.

All other points are great.

[WebBreacher]

I see your points @AccentuSoft. It'd be simple to change the boolean T/F value to "insert the characters that cause issues for this site". I do see it as more complete and customizable too. Will make the change to the proposed list above. Thanks.

[TURROKS]

Looks good to me as well. I would need to slightly modify the code to account for the change on the valid parameter, but other than that, it should be fine. Thanks for the heads up Micah.

[WebBreacher]

This has been completed.

[smicallef]

SpiderFoot has been updated to reflect this change.