Regular expressions not working in json-path

[hognala]

This is a duplicate of #61406, which was closed because it was stale (I couldn't reopen it)

What happened:
Trying to use the following json-path expression:
kubectl get secrets -o jsonpath='{.items[?(@.metadata.name=~/^test$/)].metadata.name}'
resulted in the following error:
error: error parsing jsonpath {.items[?(@.metadata.name=~/^test$/)].metadata.name}, unrecognized character in action: U+007E '~'

What you expected to happen:
Getting the list of secret names that match the regular expression.
How to reproduce it (as minimally and precisely as possible):
kubectl get secrets -o jsonpath='{.items[?(@.metadata.name=~/^default-/)]}'
Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.10.2
• Cloud provider or hardware configuration: Dell PC
• OS (e.g. from /etc/os-release): Ubuntu 18.04.1 LTS
• Kernel (e.g. uname -a): Linux pwt-test-cluster-master 4.15.0-34-generic Expanding guestbook documentation #37-Ubuntu SMP Mon Aug 27 15:21:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Install tools:
• Others:

/kind feature

[hognala]

/sig cli

[liggitt]

regular expressions are not yet supported in the jsonpath evaluator, marking this as a feature. since this is also used server-side, we need to make sure any addition doesn't open allow pathological expressions to DOS the apiserver

[ashishranjan738]

i would like work on it @liggitt maybe as a GSOC project too.

[soltysh]

@ashishranjan738 go for it

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[DTMad]

This is still a problem - please fix this

[krejkrejkrej]

/remove-lifecycle rotten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[eduardobaitello]

/remove-lifecycle rotten

[krishofmans]

Also ran into this issue, assuming since there's an -o jsonPath that it would be a complete implementation of jsonPath ... apparently it's not.

[zetas]

Same here. Just been banging my head against this for hours. It really should at least say on the json path kubectl docs that it isn't fully supported.

[povsister]

Ran into the same issue.
At least there should be a WARNING on document saying that jsonpath features are not fully implemented.
Apparently I got nothing from doc and it wasted me 20 minutes checking jsonpath doc to see if I am wrong with syntax.

[ozbillwang]

@ashishranjan738

Any updates?

[lavalamp]

The below is actually shorter. You can also make a jq script to do pretty much anything and even maintain it as a file if you wish. But in this case grep is very simple.

kubectl get secrets -o json | jq '.items[].metadata.name' | grep '^test-.*'

I only object to jsonpath being in kubectl a little bit, but I would object greatly to it being promoted to an actual api.

[povsister]

The below is actually shorter. You can also make a jq script to do pretty much anything and even maintain it as a file if you wish. But in this case grep is very simple.

kubectl get secrets -o json | jq '.items[].metadata.name' | grep '^test-.*'

jq is not installed by default on several Linux distributions.
For maximum compatibility and migration, using jq to achieve simple purpose is not a good idea.
Especially for enterprise users, generally their machines can not access Internet. Making it hard to install additional dependency.

What's more.
If my purpose is not to find all pods names start with test-, but to find all images used by pods whose name is test-* ?
Oops.. You need a for loop and lots of additional shell codes to do so if there is no jsonpath regex support.
That's ugly and painful.

I only object to jsonpath being in kubectl a little bit, but I would object greatly to it being promoted to an actual api.

JSONPath is widely used on client-side. I prefer it to be a useful tool instead of a concrete API.

For now, I usually use kubectl with jsonpath in shell scripts to fetch a batch of desired resource names/template specs.. etc. Then the result can be directly used in for loops to do something bigger.
It saves me a lot in handling error. One command is dependency-free and obviously better than 3 or 4 command pipes. Isn't it ?

[survivant]

I just found this issue after looking for hours. I'm not alone :)

we should have to rely on external command like : grep, awk, jq...

so as my understanding.. we can't find with regex in jsonpath for now.

I was trying to find all pods starting with name : test* (I wanted to add a label only on those pods)

[reschex]

Whilst I agree with the rest of the points, I want to dispell this notion before anybody else sees it and gives up on jq:

What's more.
If my purpose is not to find all pods names start with test-, but to find all images used by pods whose name is test-* ?
Oops.. You need a for loop and lots of additional shell codes to do so if there is no jsonpath regex support.
That's ugly and painful.

This is a perfectly valid use case for jq and doesn't require any further for loops or any other shell code for that matter.

k get po -o json | jq -r '.items[] | select(.metadata.name | test("test-")).spec.containers[].image'

[webmutation]

This is very annoying... If JSONPath does not support regex, then it is not very useful, and should probably be removed altogether.

I fully agree with @povsister jq is not present in many linux and windows distributions, creating an additional dependency and in our case we can't even install it since we don't have admin rights to install it. and in some cases grep is also not available only kubectl.

Also you are wasting a lot of developers time, that normally use JSONPath and discover this issue, at least put a note or something in the documentation. https://kubernetes.io/docs/reference/kubectl/jsonpath/

[lavalamp]

If kubectl is more commonly available than grep then we have really made it :)

We'd be happy to accept a PR improving the documentation.

[povsister]

Actually, there is no RFC standard, and even no IETF draft for JSONPath syntax.
Till now almost every implement of JSONPath is based on "unofficial" and old Stefan Goener JSONPath, with some arbitrary modification. (You guess what, regex support is also "arbitrary", because it does not have any standard reference).

Non-standard behavior will cause many long-lasting or legacy problems to application lifecycle and project maintainers which is strongly discouraged.

So, like JSON Strategic Merge Patch(which is a K8S modified version of JSON merge patch), if we want to extend JSONPath implement in K8S, the first thing is to raise a standard reference called Kubernetes Enhancement Proposal(KEP). I have already done this part in kubernetes/enhancements#1887
It describes the past and future needs of JSONPath, also with a completed syntax reference book. But associated reviewer does not have enough time and effort for this proposal, seems we need more time to discuss it.

[webmutation]

@lavalamp To give context, In the European Commission we (5.000+ developers) have Windows 10 Corporate laptops with no admin rights. That is why grep is not available, but kubectl is available as a virtualized app (App-V) that we can install. We cannot install jq (it has to be requested and an approval process that takes days or weeks), same for mingw it takes time to have it approved since it is not considered a standard request.

So as you can see having JSONPath regex implemented in the kubectl cli would make our lives a lot simpler, because having to request new utilities just to handle exceptions is a PITA. I don't think we are the only audience of corporate users that have this kind of restriction.

Very well, I will make a PR for the documentation... I think it will save others the same frustrations. However I don't see why this is not added to the cli, I mean kustomize was added to the CLI which is a more invasive and opinionated decision than to support JSONPath using Go style RegEx.

Any developer understands RegEx and tooling is simplified making the CLI more useful and less dependent on tooling that only creates drift, because some may use jq, others may use fx or jello or any other json library. Then they have to use the right version of jq... in our case we had to write python scripts to do something in 10 lines that could be done in 1 using the CLI. Its not the end of the world, its just frustrating and affects developer ergonomics, you are developers as well, so when a tool makes you do workarounds I think the feeling of frustration and annoyance is similar. I only mention this because there is already a PR by @povister that could make this problem go away.

[krishofmans]

@webmutation while I agree with your general point, I disagree that the factor of an organisation that more or less actively tries to hinder development teams by denying them tools to properly do their job should influence any decision. I know there are plenty of organisations like that, but they should just bleed money for the fact that they prevent people from doing their job for that. Not influence outcomes of PR's :)

In case of the EU they will bleed money because of this, and it's tax payers money, so they will also never fix their organisation.

[webmutation]

@krishofmans the point here was to explain why it is not easy to install a lot of tools in corporate environments, it is just one scenario. Just to give context to @lavalamp comment that it very strange that we would have kubectl but not grep, it may seem crazy, but its true, this type of situations exist. In particular in corporate environments. I have colleges that work also on financial and telecom institutions with this type of restrictions. It's 💩

I fully agree with you and I fight/negotiate everyday to improve things... currently we are not even allowed to install docker in development laptops, but we have Corporate PKS and we can run there (something that took 5 years of negotiation). Luckily we have AWS environments where we have freedom, but deployment environments (ACC, PRD) are hosted on DC, a lot of restrictions in the name of security, so we are limited.

Anyway, the point that I try to make and nevermind the context (corporate or not), is that one tool is better than many tools from a developer ergonomics point of view. And in particular its better than having a half implemented feature like JSONPath with no RegEx... Since there is already a PR and the reason seems to be its not standard, but there is no JSONPath standard currently supported, it would make more sense to support Go style RegEx, since most of Kubernetes community is used to it and document the usage.

[lavalamp]

Don't get me wrong, that sounds like a super frustrating situation to be in, I'm sorry you have to live with that. (This is a horrible hack and probably not approved by corporate overlords, but, if you're allowed to run things in a cluster, you could run grep or jq in a container via kubectl exec and/or attach!)

My own concern is ecosystem fragmentation, I don't want to run a variation of JSONPath that ends up not being standard. E.g., if we go with go regexps and then some other flavor is standardized... then we would have a major problem: no course of action would make both existing and new users happy.

That said, this is only kubectl we're talking about, not the API. So, it's actually SIG CLI members that should be the gatekeeper here, not me.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[krejkrejkrej]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[krejkrejkrej]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten

[k8s-triage-robot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-contributor-experience at kubernetes/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gmile]

For what it's worth, JSONPath is being standardised at the moment. The work can be tracked here: https://datatracker.ietf.org/doc/draft-ietf-jsonpath-base/