-
Notifications
You must be signed in to change notification settings - Fork 71
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Kamelet for ElasticSearch Sink provides no mechanism to set indexId or ce-indexId #673
Comments
Well, the per-kamelet docs are completely automatically generated from the yml kamelet descriptor. I share your frustration. Writing the latest version of the docs generation I was hoping to understand what could be customized in a kamelet and am still completely mystified. |
The documentation needs to be improved. Contributions are always welcome. This is an open source project, anyone could help. What is unacceptable is reading something like the generated docs is useless, this is really unpleasant to read for the contributors working on this. I'll try to provide an example tomorrow or next week for this particular case. |
David, probably you don't have an idea of how customizing a kamelet because you're only focusing on docs and website. Maybe if you ever try to run a real kamelets you will have an initial idea of what you're talking about. |
OK, so I read https://camel.apache.org/camel-k/next/kamelets/kamelets-user.html and have perhaps some clues and some questions.
I also think the description of extract field action is pretty close to useless.... "Extract a field from the body".... and then what? Trying to find the source code behind this one, it appears to be in Personally, when someone says the docs are useless, I tend to interpret it as "I'm so frustrated I can't figure out how to do what I want!!!" and try to respond to that. |
I found camel-kamelets-utils.... I had my search set wrong in IDEA. I'll look into what the extract field action is doing more tomorrow. |
The However if you look at the kamelet-component document, I think it is pretty easy to understand what the various actions types are meant to be used as they are pretty much like any other component/eip. You are very welcome to improve the documentation to include a paragraph about actions.
It would be nice to report what the various sources and sinks are missing. Bear in mind that
I agree that in general there should be a better description of what an action is doing but about the
The code is available in this repository, according to the doc dependencies, the |
David, you're not trying to do that, you're always trying to find faults in how this PMC work. This is the only thing frustrating, nothing else |
I'm glad this issue has sparked a lot of conversation. Consider this some documentation feedback.
Camel-K developers: You've done an excellent job with the product. I should point out that you'll get much more constructive feedback and increased usage if you bring the documentation up to minimum viable product level. The user community would love to provide constructive feedback but we are working with what appears to be a lot of barriers to success and its due primarily to not being able to understand how these yaml files (KameletBindings) can be used to get something done. I'll say it again
is not adequate documentation especially for a product that boasts releasing version 3.11.5 on 31 Dec 2021. But the above is ALL the documentation there is on this capability. Disheartening as it is to hear, the documentation sucks. What are appropriate values for "field"? What are legal values for "value"? Is there a way to pull values from the cloud event headers? Is there a syntax you're abiding by to specify where in a json payload the 'value' will be placed? Is there a templating tool to bring some intelligence to "The Field" and "The Value" strings? Can I pull values from previous steps of the kameletbinding? If so, how? Is there more to the syntax than what is shown in the KameletBinding exemplars? If so, where is it documented? What can we do? What functionality is present? What can we not do? What functionality is missing or broken? Basically, if the feature is not complete, please help us help you. Tell us what's missing, what's broken, what's planned. Include a link in the documentation for the feature if not to the specific code implementing the feature, then at least to the repo containing the code. A note on your documentation audience: Users coming to Kamelets from a Knative/CloudEvent/Kamel Operator perspective interact with your product via resources deployed into a K8s cluster. Listing language specific dependencies in this scenario is a bad code smell. What value is there in telling your audience about your various Java dependencies for your code? It may be of value to people writing more in-depth camel integrations or raw custom kamelets but in general your implementation details should not be a concern to people consuming the solution at the K8s integration tier. Does any of that dependency info in the docs result in something actionable by camel-k users? I submit that your documentation should focus on how to use all of the nifty things you've built into those Kamelets used in KameletBinding files (the semantics) instead of just what makes a syntactically correct yaml file. With respect, Reggie |
This is not a product. It's an open source project. Also there is no camel-k release 3.11.5. What you are refer too is plain camel version, and the documentation you need to refer to is different. Again there is no product here. You're using something for free here. So you need to respect the work done here. |
For a user POV what matter are the properties the kamelet exposes, like for any camel endpoint. Then as for each artifact, a kamelet CR come with some additional metadata which in this case are the dependencies and routes. Such metadata is needed by the machinery, like the camel-k operator, tooling and other stuffs to work but no one expects users to look at them. Hope this clarifies at least this aspect. |
SOLUTION?I have devised a solution to my basic underlying problem - When we sink to ElasticSearch, instead of hardcoding an indexId as the default if none is present, we remove the "otherwise" clause in the elasticsearch-index-sink Kamelet. WAS:
REMOVE:
RESULTING IN:
This allows a blank indexId to be passed to ElasticSearch allowing it to autogenerate an indexId. @oscerd I hope this is a reasonable mod to the Kamelet. Sincerely, Reggie |
From my testing and reading of the code at camel-kamelets/library/camel-kamelets-utils/src/main/java/org/apache/camel/kamelets/utils/transform/ extract-field-action - Replaces the json cloud-event payload with the contents of an entry in that payload. It's not clear what syntax to use to step into a json object/array to target a particular nested or array based entry. (JQ?)
insert-field-action - Adds a hard coded field:value to a json cloud-event payload - its not clear what syntax is used to specify some location in the json payload where the field should be inserted. (JQ?)
insert-header-action - Adds a hard coded header:value to a cloud-event header. There does not appear to be a bean associated with this so its implementation is a little harder to find.
??BUG??The value fields in the above state they are constant values. One would assume that any legal json string would be valid for the "value". This is not the case. There is some (inappropriate?) parsing of the value field. Strings like "{{ xxx }}" and "${ xyzzy }" should be valid values but cause a failure in the integration. If one understands what's going on with parsing, one could inject malicious code blocks into that expansion - I haven't found all of the implementing code so I do not know the extent to what gets parsed and how. |
The fix for es index is already done 9ffbe77#diff-da079f7d2f69897015758cfb8d5a1b24dc3ba677b559c2c3cafb29042c9bb558 |
It will be available in 0.7.0 |
I have been a huge fan and promoter of Camel-K since I discovered it approximately 1 year ago. It's a very useful tool and can dramatically reduce large system complexity. I'm more than aware of the alpha / pre-alpha quality of some of the components. As such, I've tried to work around the various shortcomings without filing bugs. @oscerd thank you for making this repair. When the community says the documentation useless, don't take offense, sometimes the truth hurts. Take it as an opportunity to make things clearer for the community. The documentation around KameletBindings as described in the Kamelet Catalog fails to provide sufficient detail to use the software effectively. I hope I provided some suggestions that clarify what these actions do so that it's not so confusing to others. The Open Source developers have a dramatically better insight into what's going on under the hood. As consumers of the solution, we often do not have insight into the inner workings of the code. This makes it impossible to provide constructive feedback. Many users of this technology have no Java background and digging through code ends up being a fruitless effort for them. The only recourse is to file a bug/open an issue. I've been working this problem for at least 6 months and have consumed 100's of hours of time and resources on this. Feedback in this issue chain helped me find the code that lead to better understanding of what these actions are doing. I hope my comments and suggestions lead to updated documentation and code that is better for all who wish to use this open source product. By the way: If you're not already aware. All of the links in the Kamelet Catalog that point to the Kamelet Source Files are broken and lead to a 404 on GitHub. On a personal note: I wish you had responded with this latest bit of information about a fix 20 hours ago instead of waiting for me to discover the same or similar solution before telling us that there is already a solution developed 17 days ago. I was busy trying to find a solution based on combining and working with actions documented in the Kamelet Catalog. Now I know, having read some of the code, that there is likely no solution to the problem by attempting to manipulate the configuration of one or more actions in a kameletbinding. |
You weren't asking for this fix, but for something else, using the extract action for setting the indexId from the json routing in the integration. This is not a solution, it's just a workaround. Thanks for reporting about the links, I'll take care of fixing them. Maybe there were some updates when we changed the way the documentation generation with some PRs from David. I will always speak up when the work from this community and from the oldest contributors will be underrated and I really wish that members of this PMC do the same, because in the last weeks I just seen attempts to criticize the work done and find faults in what it is done (if someone feel I'm talking about him, yes, I'm talking about you, David). 20 hours ago I couldn't answer about the fix, because your problem was and it's still different. The 100's hours of time you consumed, it's 500 hours of work done from contributors around Kamelets and camel-k. The fact that you don't know the code or you have troubles finding where the code is or how it is working, it's part of the game. Camel is a really complex ecosystem, documenting everything will always be impossible, given the number of real contributors working regularly and with passion on this. I'll take comments like yours as offence, because you and many others, consider, ALWAYS, our time for granted, which it's wrong and unpleasant to read. That say, thanks for your feedback and I'll try to improve the upstream documentation, in my spare time, as always. |
Thanks for fixing the source links. I don’t see an update for 0.5.x? I’m a bit surprised the link checker didn’t find the problems.
I’m really glad to discover my fears about the kamelet dependencies linking to unreleased code are unfounded, although since the 0.6.x branch doesn’t appear to have been updated after release the kamelet dependencies on the branch currently do point to snapshots… but the poms also say the version is main-snapshot.
I apologize for repeatedly discovering something about the processes here that worries me and, after not being able to discover what the actual process is, expressing my doubts that it’s in line with apache polices. I can see that that’s probably rather infuriating. I’ll see if I can find another way of asking for information; what I’ve been doing is bad for everyone.
If there’s something else I’ve said that you are offended about, please let me know what it is. I’m trying to contribute, not cause problems.
I think camel is a pretty amazing thing, which is why I’m spending so much time working on the docs and trying to get them to make the relationship between the different parts clearer.
David Jencks
… On Jan 6, 2022, at 9:46 PM, Andrea Cosentino ***@***.***> wrote:
You weren't asking for this fix, but for something else, using the extract action for setting the indexId from the json routing in the integration. This is not a solution, it's just a workaround.
Thanks for reporting about the links, I'll take care of fixing them. Maybe there were some updates when we changed the way the documentation generation with some PRs from David.
I will always speak up when the work from this community and from the oldest contributors will be underrated and I really wish that members of this PMC do the same, because in the last weeks I just seen attempts to criticize the work done and find faults in what it is done (if someone feel I'm talking about him, yes, I'm talking about you, David).
20 hours ago I couldn't answer about the fix, because your problem was and it's still different.
The 100's hours of time you consumed, it's 500 hours of work done from contributors around Kamelets and camel-k. The fact that you don't know the code or you have troubles finding where the code is or how it is working, it's part of the game. Camel is a really complex ecosystem, documenting everything will always be impossible, given the number of real contributors working regularly and with passion on this.
I'll take comments like yours as offence, because you and many others, consider, ALWAYS, our time for granted, which it's wrong and unpleasant to read.
That say, thanks for your feedback and I'll trying to improve the upstream documentation, in my spare time, as always.
—
Reply to this email directly, view it on GitHub <#673 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAELDXXJQ5NBY6C4NA4RGODUUZ42HANCNFSM5LLAMB2A>.
You are receiving this because you commented.
|
On 0.5.x the kamelets where still in the main folder, so links should be ok. |
Yes, but the version is wrong, it should be `https://github.com/apache/camel-kamelets/blob/0.5.x/{name}.kamelet.yaml` <https://github.com/apache/camel-kamelets/blob/0.5.x/%7Bname%7D.kamelet.yaml%60>
It’s too late here to make a PR, but if you don’t get to it I’ll do it first thing tomorrow.
The version is wrong in 0.6.x also.
David Jencks
… On Jan 6, 2022, at 11:47 PM, Andrea Cosentino ***@***.***> wrote:
On 0.5.x the kamelets where still in the main folder, so links should be ok.
—
Reply to this email directly, view it on GitHub <#673 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAELDXXQBIS37L64EQBLRMTUU2LAJANCNFSM5LLAMB2A>.
You are receiving this because you commented.
|
I'm not personally offended by anything you said. My aim is to defend the work we're doing here every day, because I do believe Camel exists for his contributors, occasional contributors or regular contributors is the same. If I would be a newcomer in the open source world and I would read some of the comments in this issue about documentation that sucks, people from the PMC saying "yeah, right, I'm frustrated" etc. I would stay away, because it looks toxic and it's not. You're right about 0.5.x, I'll fix it there too, thanks. |
Yeah the page already refers to the developer guide. Adding a paragraph prior pointing to the user guide is a good idea. And you are surely welcome to send a PR.
Actions was added later, we started with sink/sources. You are welcome to create a ticket so the task is not forgotten.
David, that is a harsh statement to call it "toys" - Kamelets are in heavy development and they are on 0.x release (not reached a 1.0 GA yet). Sours and Sinks are the most important goals of Kamelets, to make it even easier to connect to/from systems. Take a moment to read also some of the blogs about Kamelets that we have published, such as
David, this is again a harsh statement about "violation of XXX". As you have learned now there is a reason for this as explained by Luca.
Okay but you did not only do that .. you also made acquisitions and harsh remarks about the camel-kamelets project, which made this ticket worse and causing frustrations among the maintainers of this project. |
With #681 You should be able to set headerOutput as parameter and you'll get whatever value you want to extract into the CamelKameletsExtractFieldName header. So you could use the header in the subsequent steps. I'm going through the actions for updating and improving some information |
With a little update to insert-header action you should be able to do something like
|
This will be available in 0.7.0 |
@oscerd Thanks for all of the effort, I think this mod adds a huge amount of capability. But lots of questions and some suggestions: I'm curious about what it implies to use "Simple" and not Constant for the value parameter binding. This would be very handy in the docs. At a high level, I think it allows for strings with embedded expansions "${some_variable}" but that's just an assumption on my part. Could you add into the documentation what variables we have access to in this expansion and their types (Map, Array, Object, etc.) , e.g. I see a potential problem down the road with using a single fixed header (CamelKameletsExtractFieldName). What happens when I need to extract more than one thing to be placed in the header? Could you do something like this? Note the "field" as well as alternate use of "headerOutput". If we know the payload is JSON, then "field" needs to be interpreted to identify any entry in that json. Ideally this would use a common syntax such as that used by JQ B.T.W. what happens if the thing requested in "field" is not a simple type (string, boolean, number)?
If my understanding of "Simple" is correct, then in a KameletBinding one could establish the headerOutputs in steps A and B and then combine the headers in a subsequent step C as per Does the Camel DSL or some subset extend to KameletBindings or is it limited to Kamelets? How much room for configuration do people have if all they want to do is blend existing Kamelets into a solution? SPECULATION: |
I added this to the page |
You could have an idea of how simple works and what is supported from here: https://camel.apache.org/components/3.14.x/languages/simple-language.html I can add a link in the description about simple language and maybe add some examples somewhere. Yeah.
For the header name it makes sense, need to think about a possibile good way of sanitizing. I'd probably also make the sanitize logic optional, like having headerExistenceCheck true or false. Not sure. I think you're meaning to concatenate more extract-field-action steps and then concatenate the headers when you've done in a new header to use down the route. In that case, yes, it's not a good design what I've done, because you'll end up to have the same header overriden. Ok, I'll check this scenario too.
Well, if you have something like a complex type, what you'll get will always be a JSON (just a subset of the original one), with that, you could for example use deserialize-action and transform the JSON into a POJO instance. But in the original design, what is going through actions is always a JSON. By the way maybe adding some examples will be better.
Yes, you could do something like that, but first we need to sanitize the headerName in some way.
In the beginning Kamelet were only part of camel-k, so you could use the kameletBinding as part of a camel-k integration. Later we added the support of kamelet into a Jbang extension, still experimental. So you can use both camel-k, through KameletBinding yaml or you can use camel-kamelet-main through a Camel yaml route using the camel-kamelet component or you could even use the camel-jbang extension for a fast feedback about the yaml route based on kamelets from catalog you're using. But this needs to be defined in the documentation of catalog, camel-kamelets, camel-kamelet-main and camel-jbang a bit better.
There were an issue in Camel JIRA about a camel-cloudevent compoonent. But I don't remember if we had some developments around this. |
No, the goal of KameletBinding (which should be renamed to Binding) is to abstract the underlying engine so we do not expect to add any Camel DSL capability directly to the Binding, if you need such flexibility then you should think about using an Integration where you can use any Camel DSL.
Adding scripting capabilities through a Kamelet is pretty simple. But I want to point you that this has almost nothing to do with CloudEvents as a CloudEventn is nothing more than an envelope that matches what in Camel is a Message (as Andrea pointed out, there is an issue about a better CliudEvent support in Camel). Camel offers already a way to manipulate Messages (of course) by using a scripting language. |
Greetings,
I've got a Kamelet that sources a KafkaTopic and sinks to ElasticSearch.
The cloud event payload is JSON.
The payload does contain a unique identifier that I want to use as indexId, I cannot figure out how to use "insert-field-action" or "insert-header-action" to inject indexId/ce-indexId with a unique value from the payload (or for that matter ce-attributes). Attempts to use "extract-field-action" might be of use but there does not appear to be any way to actually use what was extracted. (I was hoping I could extract a field and then use the contents to insert a header or insert a field).
Frankly the Kamelet documentation is useless. It provides no data other than boilerplate as if automatically generated. No explanation of fields, their use, or meaningful examples. See the next three code blocks as exemplars:
or
or
Do I have to roll my own Kamelet to do something so obviously simple?
I must be missing something.
Thanks,
Reggie
The text was updated successfully, but these errors were encountered: