FR: Automatically remove common Cache Busters

[CoreyD97]

The tool performs great, but sometimes I find apps that end up with a ton of duplicate resources since the URLs often contain a cache buster. This is typically the _ or v parameter with its value set to a unix timestamp. It'd be great to automatically ignore these parameters.

Thanks again!

[xnl-h4ck3r]

Hi @CoreyD97. Have you got some examples you can share? If there are a number of URLs with similar parameters, it should take care of those, so maybe I'm misunderstanding what you mean.

[CoreyD97]

Sure! I've hidden the target here but can share the domain if you need. The input file was taken directly from the waymore output.

[xnl-h4ck3r]

Thanks. I'll look at adding some option to provide a list of parameters to exclude

[xnl-h4ck3r]

Just to clarify... Were you wanting cache buster parameters to be removed completely from the output?
So if you had this input:

https://test.com//jquery.paginate.min.js?v=2022090506&_=1558677431432
https://test.com/jquery.paginate.min.js?v=2022090507
https://test.com//jquery.paginate.min.js?v=2022090507&_=1558677431434

You just want to see this as output?

https://test.com//jquery.paginate.min.js

Also, just so you are aware, by default jquery is in the FILTER_KEYWORDS so a lot of those URLs wouldn't appear at all if you removed the parameters _ and v. The FILTER_KEYWORDS are only applied to URLs without parameters.

[CoreyD97]

Those are just some examples, but yes that's the expected outcome. That particular website ended up with a bunch of duplicate entries because of the presence of these cache buster parameters. As for the FILTER_KEYWORDS, that behavior is fine :)

[xnl-h4ck3r]

Great! I have done the change and almost ready to release. I have added a REMOVE_PARAMS key to config.yml and also added an argument -rp/--remove-params that can override the config file. Both will be a comma separated list of parameter names.
I was thinking of defaulting the config key value with cachebuster.
I think it's best if the parameter names are case sensitive, so maybe I can set the default as cachebuster,cacheBuster so people can see an example if they want to add more param names. What do you think? Any parameter names you'd add?

[xnl-h4ck3r]

I kept it as cachebuster,cacheBuster as the default :)
You can now get that change in v2.0, the latest.
Thanks for using it and taking the time to raise suggestions!