support underscores in hostnames #400
Comments
cooniur
changed the title
|
Thank you for reporting this, but this issue was already patched here and released in v2.5.0 of the plugin. Logstash 2.2.0 shipped with v2.4.1 of the plugin. If you're not running the latest version of the plugin (which is presently 2.5.5), you should update it by running: ...from the directory where Logstash was installed. |
|
@untergeek I'm glad to hear that the bug has been fixed! Will do the update. Thanks! |
|
Hey @untergeek , here is a quick feedback: after upgrading, I got this error while starting Logstash: And I noticed there is a |
|
Another quick feedback: Even after I upgraded to logstash 2.2.2, I still get error with these configs Elasticsearch: 1.5.0 Config 1Running Error is: Config 2Running Error is: Ping the DNS name: |
|
Please report the output of: |
|
Output is: |
|
We should still try to upgrade to 2.5.5, which was released today. Please run the If it persists, we'll have to update the logic to allow long, multi-level subdomain syntax. |
|
Tried to update, it gives some error: Although, And when I tried to start Logstash, it gives error: I'll let you guys handle this issue. Please re-open the ticket. |
|
We'll see what's going on. |
|
Thank you! |
|
I'm having the same problem |
|
I can't specify a valid hostname in the config This happens with the latest version of Logstash I am testing this with with docker: even older versions of logstash fail The hostname and connection to it is working |
|
the problem seems to be that the expression in https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/http_client.rb#L132 does not accept underscores - which by the way is correct according to the RFC. |
|
Hi there. I have the same problem. ¿ So we have to limit our dns resolution to the hostname RFC ? ¿ there any possibility to add a 'url' field instead of hosts list ? Im currently working with consul , and the service name has "_" in its name. |
|
For a quick workaround until the issue is officially resolved, you can permit underscores by patching the following two files:
Modify this regular expression to allow '_' as well:
Modify this regexp as well to allow '_':
|
|
Regarding a potential fix, could we include the addressable gem and call |
jsvd
changed the title
|
Quick solution for the ones who use |
|
https://bugs.ruby-lang.org/issues/8241 is related if you'd like to go down the rabbit hole of why Ruby doesn't like the underscore. My solution was to set up a CNAME DNS record without the underscore, which points to the DNS name with the underscore. |
|
For anyone using the logstash:5.0.1 Docker image here's the RUN command you can put into your Dockerfile to patch both files: |
|
@bryanspaulding: on 5.1.1 this throws What gives? |
|
this seems to be a characteristic of ruby's 1.9.3 URI parsing library, which jruby-1.7 conforms to: |
|
One alternative is to use the addressable gem's uri parser. this gem is already included in the logstash release and seems to conform better with the rfcs: @andrewvc thoughts? |
|
I'm running into the same issue, using logstash 5.0.2 |
|
Hi I'm running into the same issue too. I was wondering if there is any plan to fix this. |
|
Hi @jsvd , it's been quite a while. Any update to this issue? |
|
This is still an issue. We have since moved the code that deals with URI to the native java classes, but they also have troubles dealing with hostnames with underscores (which goes against the rfc952): |
|
https://bugs.openjdk.java.net/browse/JDK-8170265 was closed "Not an issue" with the implication that we must parse this ourselves and call We're probably on our own, here. We'll need to parse the URL external to java.net.URI and create the URI/URL instance ourselves :\ |
|
@jsvd let's see if we can reopen the JDK issue. I don't see how to create an account on that jira instance, though. Can you drive this? |
|
Yes. one option could be considering using the java.net.URL instead of URI. the URL class for some reason accepts the host with underscores correctly: OTOH there's a hack: URI uriObj = new URI("https://pmi_artifacts_prod.s3.amazonaws.com");
if (uriObj.getHost() == null) {
final Field hostField = URI.class.getDeclaredField("host");
hostField.setAccessible(true);
hostField.set(uriObj, "pmi_artifacts_prod.s3.amazonaws.com");
}These two "solutions" aside, from what I understand there's a pretty high bar to enter the jdk jira. For the mortals, the entry point I see is http://bugreport.java.com/, I'll start there. |
|
Hi, it seems, this issue is still not fixed in logstash 6.1.2, are there any updates? DNS specification specifically allows underscore and in fact they are very common. |
|
Hi, this issue is still not fixed in logstash 6.3.0. |
|
Issue still present on logstash 6.4.0 |
|
What the heck - just encountered this issue on Docker Swarm, where underscore in hostname is automatic when using stack deploy - and found this issue from 2016! What to do? : ( |
|
It's logstash 6.6 and this is not yet fixed...guys this is a bad joke...we are living in the container world...no one wants to type fixed IPs !!! |
|
This link might be useful: docker/compose#229 Seems like docker's not moving on this, despite |
|
The confusion probably comes I think from the fact that In practice, I see |
|
Pure nonsense that this this is still an issue after 2 years. Using Docker Swarm where underscores are appended after the stack name. @kevin-bennett-ags, one way to resolve is to install dnsutils, add a known string for the elasticsearch host name (e.g. ELASTICSEARCH_HOST) in the output.conf file, and create a bootstrap file with the following command: sed -i "s|ELASTICSEARCH_HOST|$(dig +short )|g" /etc/logstash/conf.d/30-output.conf |
|
Issue still present on logstash 7.2.0. However, Elasticsearch 7.2.0 and Kibana 7.2.0 support underscores in hostname. I'm confused about this issue, it doesn't be mentioned even in document. |
|
Issue still persist in Logstash 7.4 and plugin logstash-output-elasticsearch (10.1.0). Results with: |
|
@athlan Try remiving "http://", just use hosts => ["elasticsearch.infra_network:9200"] |
|
@diegorosano seems that scheme must be defined, now error says: Illegal character in scheme name at index 19: elasticsearch.infra_network:9200 Also, issue still occurs on 7.4 |
|
Still in Logstash 7.6 Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: Illegal character in scheme name at index 7: ansible_es01:9200" Filter output { |
|
deploying a stack into docker swarm with docker-compose.yml, the service name is stackname_servicename. But, using Docker 18.09.9 I tried using just the service name in my logstash 7.6 config and it worked fine. So, using the default ingress overlay network, you can refer to another service container from within a service container by either its full service name (mystack_myservice) or by just the service (myservice) which avoids the underscore problem |
|
TL/DR: Logstash 7.6.2 doesn't like hostnames with underscores, change your hostnames. I'm running logstash (7.6.2), elastic and filebeat in docker using docker compose, with the usual setup: filebeat --> logstash --> elasticsearch I had the same problem with the hostname in logstash. My container names were defined with underscored in docker-compose.yml and I was using a common network so each container has access to the other by using the container name as the hostname. So filebeat connects to logstash just fine using underscore hostnames, elasticsearch starts up fine with underscores. It turns out this logstash plugin has a problem with underscored hostnames. Once I changed everything from |
|
TL/DR: Logstash is implementing the correct host naming conventions expressed in RFCs, etc. These do not allow underscores. In later years these have been ignored and people have implemented stuff using underscores where they were not supposed to be. I have come up with a workaround (I'm not sure if this has been talked about before in here), in which I define a custom pattern definition, local to the matching pattern that I need to enable for these hostnames. This could be a potential workaround, YMMV. Standard disclaimers apply: |
|
For those of you looking to do this in Docker Swarm, docker's internal DNS has a From any docker service within a single stack, you can find the ip addresses of other docker services within that same stack with docker-stack.yml pipelines/sample.conf The service name for all my elasticsearch nodes is named Reference: |
@jnovack While your answer about the internal DNS with I'm experiencing this issue now, and my logstash service is finding with |
|
Hi #!/bin/bash
SERVICES=(
"management-nodes-elastic_shahriar-es-node-1"
"management-nodes-elastic_shahriar-es-node-2"
"management-nodes-elastic_shahriar-es-node-3"
)
for i in ${SERVICES[@]};do
RESPONSE=$(ping $i -c 1)
HOSTNAME=$(echo "$RESPONSE" | head -n 1 | grep -oP "(?<=PING ).[^\(].*(?= \()" | sed "s/_/-/g")
IP=$(echo "$RESPONSE" | head -n 1 | grep -m 1 -oP "(?<=\().*?(?=\) )" | head -n 1)
grep -qxF "$IP $HOSTNAME" /etc/hosts || echo "$IP $HOSTNAME" >> /etc/hosts
done
then also if you use docker-compose file you must change Logstash entrypoint to this: version: '3.8'
services:
logstash:
image: logstash:8.2.2
configs:
- source: logstash-configs
target: /usr/share/logstash/config/logstash.yml
- source: service-resolver
target: /usr/share/logstash/service-resolver.sh
entrypoint: bash -c "apt update -y && \
apt install iputils-ping -y && \
/bin/bash /usr/share/logstash/service-resolver.sh &&
/bin/bash /usr/share/logstash/service-resolver.sh &&
/bin/bash /usr/share/logstash/service-resolver.sh &&
/usr/local/bin/docker-entrypoint -r" |
This solved my problem today. Thank you!!! |
and others
Elasticsearch version: 1.5.0
Logstash version: 2.2.0
OS: Ubuntu_14.04.3_LTS_HVM (
uname -a = Linux my-test-logstash-1-us-west-2 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux)Cloud platform: AWS
Details
For the following two configs, Logstash gives different errors respectively. The 3rd config works well, though, but is not practical in a cloud environment like AWS.
Config 1 (failed)
Error message:
Note the extra "
http://" in the:hosts=>field in the error message. I believe this should definitely be a bug.Config 2 (failed)
Error message:
Config 3 (passed)
This works, however, I'm in AWS environment and cannot rely on IP addresses.
The minimum requirement is that for Config 1 the parsing should work.
Please take a look at. Thanks in advance!
=================
Updated:
With Logstash 2.2.2 and logstash-output-elasticsearch 2.1, the issue still exists (see the 5th reply).
Tried to update the plugin to 2.5.5, however, the upgrade may have corrupted Logstash, and it now fails to start with error:
================
Updated 11/1/16: put the above update under the original post to avoid confusion.
The text was updated successfully, but these errors were encountered: