feat(stdlib): add parse_etld function

[esensar]

This adds parse_etld function to stdlib, based on psl crate

Fixes: #651

[esensar]

This is currently using psl crate for performance, but there is also publicsuffix crate, recommended by the same author, for cases when dynamic list is needed. psl crate is using a list that is hardcoded in the crate and autogenerated whenever it changes, meaning tracking newest list requires updating the crate.

I could switch it to that crate if it would make more sense.

Also, I was also considering adding this to parse_url function, but it might not be beneficial to that many users, so it is probably best to keep it separate. Also, it allows customizing number of additional parts provided (like eTLD+1 but for more than 1), but I am not sure if that is a valid use case. It might be better to keep it simple and just pass eTLD and domain as result. This will also always return result, even for TLDs that are not found in the list, but it will have known_suffix set to false in that case.

[johnhtodd]

Very happy to see this feature. I suspect it will have a better chance of approval if it also had documentation in the main docs repository as part of the PR.

[pront]

Thank you @esensar, this looks good!

This is currently using psl crate for performance, but there is also publicsuffix crate, recommended by the same author, for cases when dynamic list is needed.

Let's switch to publicsuffix.

Also, I was also considering adding this to parse_url function, but it might not be beneficial to that many users, so it is probably best to keep it separate.

👍

[esensar]

Let's switch to publicsuffix.

Alright, makes sense, although that might complicate things a bit. With that crate we need to download the list in runtime, so I need to figure out when to do it and how and when to cache it. I will see if I can update the PR tomorrow, but let me know if you have ideas on how I should approach that.

[pront]

Let's switch to publicsuffix.

Alright, makes sense, although that might complicate things a bit. With that crate we need to download the list in runtime, so I need to figure out when to do it and how and when to cache it. I will see if I can update the PR tomorrow, but let me know if you have ideas on how I should approach that.

Actually, this is a good point. This introduces complexity. Let me take a closer look soon.

[pront]

psl crate is using a list that is hardcoded in the crate and autogenerated whenever it changes, meaning tracking newest list requires updating the crate.

@esensar on a second thought, this is preferable to the alternative 👍

[pront]

What if by mistake we had plus_parts: 10 here? Can you add a test for this?

[esensar]

There is actually a test for that, named short_plus, but it uses plus_parts: 3 when only 2 are available. I will change it to 10 for better visibility and clarity, but this makes me think if behavior in that case is right. It will just silently return whatever is available (so "vector.dev" in the above case). Would it be better to somehow let it be known in the results that it was not able to get all of these parts, since this way it would require some additional checks on the callers end.

[pront]

but this makes me think if behavior in that case is right.

Some options to consider:

1. return the max available parts
2. return an error

I am leaning towards 1 which I believe is the current behavior but let me know if you have further thoughts.

[esensar]

I agree with case 1. That should cover the general case and keep things simple.

[pront]

Thank you @esensar!

[jszwedko]

Could we add these new functions to the benchmarks in benches/stdlib.rs?

[esensar]

Are functions like this documented in the main docs section as part of the PR typically, or is that something that the maintainers reserve as a controlled area? (https://vector.dev/docs/reference/vrl/functions/)

There is a PR open for that update and if I understood correctly, it will be merged after new VRL version is released, to be up to date with the latest available version.

[johnhtodd]

Are functions like this documented in the main docs section as part of the PR typically, or is that something that the maintainers reserve as a controlled area? (https://vector.dev/docs/reference/vrl/functions/)

There is a PR open for that update and if I understood correctly, it will be merged after new VRL version is released, to be up to date with the latest available version.

I didn't understand the method by which those pages are created, and it seems that you've already done this work - sorry for the confusion.

[esensar]

Thank you @esensar, this looks good!

This is currently using psl crate for performance, but there is also publicsuffix crate, recommended by the same author, for cases when dynamic list is needed.

Let's switch to publicsuffix.

Also, I was also considering adding this to parse_url function, but it might not be beneficial to that many users, so it is probably best to keep it separate.

👍

Sorry for bringing this topic back, but turns out custom PSL is needed. I was thinking about how to implement it and since these are not really big, it would probably be okay to keep them in memory. I thought of maybe implementing it in a similar way to how enrichment_tables are implemented, by adding additional config option to set up PSL URL (and maybe support file paths too). Does something like that make sense? I just want to check before I go into implementation. That would also mean that this function would have to move to vector repo I think.

We can also have a separate function for that "dynamic" approach, so we don't have to remove this one from vrl.

[esensar]

Thank you @esensar, this looks good!

This is currently using psl crate for performance, but there is also publicsuffix crate, recommended by the same author, for cases when dynamic list is needed.

Let's switch to publicsuffix.

Also, I was also considering adding this to parse_url function, but it might not be beneficial to that many users, so it is probably best to keep it separate.

👍

Sorry for bringing this topic back, but turns out custom PSL is needed. I was thinking about how to implement it and since these are not really big, it would probably be okay to keep them in memory. I thought of maybe implementing it in a similar way to how enrichment_tables are implemented, by adding additional config option to set up PSL URL (and maybe support file paths too). Does something like that make sense? I just want to check before I go into implementation. That would also mean that this function would have to move to vector repo I think.

We can also have a separate function for that "dynamic" approach, so we don't have to remove this one from vrl.

@pront not sure if you saw this. What do you think about this?

[johnhtodd]

Some context here: the PSL data set doesn't quite summarize things as desired, and is not helping in our goal of reducing/aggregating data to the degree that we want to be fed out of edge nodes.

I suspect almost all users of the PSL list have different interest levels of depth into the DNS tree that they wish to aggregate, based on different zone cuts. Many of the domains (voluntarily submitted to the PSL) are much "deeper" in the domain set than normally one would want to go for summarization. For example, if I'm trying to find out counters for every domain that is a registry-listed domain (or at least a close approximation.) So in the simplest case, the desire is to count how many events are inside of "amazonaws.com" as an example aggregation where we look at ETLD+1 (.com plus one zone cut to the left.) However, the PSL list shows that things like "execute-api.ap-east-1.amazonaws.com" are listed, which means that we see every zone cut underneath that zone. While they are not "ETLD+1", they act as if they are if the list is consumed without editing. Needless to say, what we wanted to have as a fairly compact set of namespace has turned out to be an exploding cardinality issue, with probably 50% of the elements in the namespace seen as un-necessarily collected based on our experience so far.

To solve this problem, we'd like to edit the PSL on our own, so that Vector uses a customized version. This doesn't seem too difficult - we're just hoping to have a URL or filepath that allows the PSL data to specified on launch. This changes very infrequently, so it does not need to be dynamic, but it does need to allow a custom file to be identified.

@pront I know this is an edge case, but it is fairly close to the heart of the concept of having the ETLD function and making it useful. Customization of data files like this I hope is consistent with Vector's focus on giving operators highly-editable functionality at the edge of the network. In our case, this turns into billions (hundreds of billions?) of events per day for us that we could condense, so it does have a significant result if implemented.

[johnhtodd]

We're kind of blocked here, and we hope that the path we're going to take isn't going to create problems with getting the patches accepted.

[pront]

👋 This thread fell through the cracks, can you summarize what is the proposed change here? Do you want add custom PSL support using enrichment tables?

[jszwedko]

Thanks for the bump @johnhtodd . I agree with @pront that the proposal isn't entirely clear in that I'm not sure what UX you are expecting.

I could imagine something like the feature to load aliases from a file that was added to parse_groks recently. That might make the function call look something like:

parse_etld(value: "example.com", psl: "/some/path/to/psl/file")

Is that something like what you were expecting? It seems like a reasonable enhancement to me, if so.

I'd suggest we limit it to local files for now rather than opening the door to loading from, for example, URLs as suggested in #669 (comment) since Vector/VRL doesn't have a precedent for that.

[jszwedko]

I think we should still keep the default psl if a custom one is not provided.

[esensar]

Thanks for the bump @johnhtodd . I agree with @pront that the proposal isn't entirely clear in that I'm not sure what UX you are expecting.

I could imagine something like the feature to load aliases from a file that was added to parse_groks recently. That might make the function call look something like:

parse_etld(value: "example.com", psl: "/some/path/to/psl/file")

Is that something like what you were expecting? It seems like a reasonable enhancement to me, if so.

I'd suggest we limit it to local files for now rather than opening the door to loading from, for example, URLs as suggested in #669 (comment) since Vector/VRL doesn't have a precedent for that.

Yeah, I guess file is okay for most cases, the URL option doesn't really offer much more, since the file can be downloaded before configuring anyways. I agree with the proposed API, but I think it would be good to move the actual file to configuration and refer to PSL by name, because that way we can parse it once on startup and then just use it, but that means that the function would have to move to vector (similar to enrichment tables functions).

I am not sure if that is alright with you, since we would have to remove it from vrl and move it to vector, since otherwise we would have to parse the file every time the function is called (since there is no caching and no runtime in vrl).

I think we should still keep the default psl if a custom one is not provided.

Agreed on that one. We can just use the library with pre-generated PSL for that I guess.

[jszwedko]

Yeah, I guess file is okay for most cases, the URL option doesn't really offer much more, since the file can be downloaded before configuring anyways. I agree with the proposed API, but I think it would be good to move the actual file to configuration and refer to PSL by name, because that way we can parse it once on startup and then just use it, but that means that the function would have to move to vector (similar to enrichment tables functions).

You can actually do this while leaving it as an argument by requiring the argument to be static and so it can be loaded only once at start-time. You can see an example of this with #194.

[esensar]

Yeah, I guess file is okay for most cases, the URL option doesn't really offer much more, since the file can be downloaded before configuring anyways. I agree with the proposed API, but I think it would be good to move the actual file to configuration and refer to PSL by name, because that way we can parse it once on startup and then just use it, but that means that the function would have to move to vector (similar to enrichment tables functions).

You can actually do this while leaving it as an argument by requiring the argument to be static and so it can be loaded only once at start-time. You can see an example of this with #194.

Oh, that makes sense, it is probably the best solution to do that then. I will then make a PR to vrl repo with these changes. Thanks!