Unable to Connect to localstack's OpenSearch with the Endpoint provided #5723

ByteProgramming · 2022-03-22T09:52:22Z

Is there an existing issue for this?

I have searched the existing issues and read the documentation

Question

I am trying to connect to LocalStack OpenSearch with the endpoint provided once domain created.
Steps: I have started localstack as docker container with the below docker-compose file

version: "3.3"
services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
    image: localstack/localstack:latest
    network_mode: bridge
    ports:
      - "4566:4566" 
      - "4564:4564"           # LocalStack Edge Proxy
      - "8080:8080"
    environment:
      - SERVICES=${SERVICES-}
      - DEBUG=1
      - DATA_DIR=/tmp/localstack/data
      - START_WEB=0
      - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR-}
      - HOST_TMP_FOLDER=${TMPDIR:-/tmp/}localstack
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "${TMPDIR:-/tmp}/localstack:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"`

Created the domain:
aws opensearch create-domain --domain-name my-demo --endpoint-url=http://localhost:4566

Result:


    "DomainStatus": {
        "DomainId": "000000000000/my-demo",
        "DomainName": "my-demo",
        "ARN": "arn:aws:es:eu-central-1:000000000000:domain/my-demo",
        "Created": true,
        "Deleted": false,
        "Endpoint": "my-demo.eu-central-1.opensearch.localhost.localstack.cloud:4566",
        "Processing": true,
        "UpgradeProcessing": false,
        "EngineVersion": "OpenSearch_1.1",
        "ClusterConfig": {
            "InstanceType": "m3.medium.search",
            "InstanceCount": 1,
            "DedicatedMasterEnabled": true,
            "ZoneAwarenessEnabled": false,
            "DedicatedMasterType": "m3.medium.search",
            "DedicatedMasterCount": 1,
            "WarmEnabled": false,
            "ColdStorageOptions": {
                "Enabled": false
            }
        },
        "EBSOptions": {
            "EBSEnabled": true,
            "VolumeType": "gp2",
            "VolumeSize": 10,
            "Iops": 0
        },
        "AccessPolicies": "",
        "SnapshotOptions": {
            "AutomatedSnapshotStartHour": 0
        },
        "CognitoOptions": {
            "Enabled": false
        },
        "EncryptionAtRestOptions": {
            "Enabled": false
        },
        "NodeToNodeEncryptionOptions": {
            "Enabled": false
        },
        "AdvancedOptions": {
            "override_main_response_version": "false",
            "rest.action.multi.allow_explicit_index": "true"
           }

Created Index:
curl -X PUT my-demo.eu-central-1.opensearch.localhost.localstack.cloud:4566/my-index

Result
{"acknowledged":true,"shards_acknowledged":true,"index":"my-index"}

Java/SpringBoot client connection code:
new RestHighLevelClient(RestClient.builder(HttpHost.create(endpoint)));

Model class for index:

@Document(indexName = "my-index")
public class MyDemoIndex {

   //private fields
}

Endpoints tried in the connection code:

http://my-demo.eu-central-1.opensearch.localhost.localstack.cloud:4566
[This gives connection timeout through java code]
http://localhost:4566
[This also gives connection timeout through java code]
http://${docker-host}/4566
[This gives 500 error.]
[Error: The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.]

Is something i am mistaking here ? or the approach could be different as I am not able to communicate with the Localstack service successfully.
My goal is to mock my opensearch service in local and should be able to connect to Localstack and communicate properly.

Anything else?

No response

The text was updated successfully, but these errors were encountered:

github-actions · 2022-03-22T09:52:35Z

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our developer guide.

alexrashed · 2022-03-22T10:35:34Z

Hi @ByteProgramming!

Does this mean that you can successfully create the index with the curl command you posted? Because if the curl is successful, it seems like your network setup should be just fine.
Since you are using the default endpoint strategy for OpenSearch (domain) the endpoint to use would be the first one.

Could you maybe elaborate a bit on what kind of client framework you are using there and what you are trying to accomplish?
Is this client code going to be executed in a Lambda in LocalStack or somehow else?
Thanks!

ByteProgramming · 2022-03-22T11:51:05Z

@alexrashed Hi Alex,
Yes when i fired the index creation command, i get this response:
{"acknowledged":true,"shards_acknowledged":true,"index":"my-index"}

What client:
I have a spring boot app. I have a config class for opensearch where i built connection to connect the opensearch.
I have a model class mapped to index.
What am i trying to achieve:
My spring boot app runs in Localstack lambda. I have localstack dynamo db. I am streaming data from dynamo db to
lambda and then writing to opensearch. Localstack lambda works fine. I am able to run Localstack dynamo. When i put
data in Localstack dynamo, it streams data to lambda. But when I tried to connect to opensearch localstack from my
localstack lambda, it gives the above mentioned error.

alexrashed · 2022-03-22T15:15:56Z

Okay, seems like the issue is the DNS resolving of the created endpoint (my-demo.eu-central-1.opensearch.localhost.localstack.cloud) in your lambda. This resolving is available in our Pro offering.

However, there might be a workaround which could work for you.
It consists of the following steps:

Change the OpenSearch endpoint strategy to path by setting the environment variable OPENSEARCH_ENDPOINT_STRATEGY=path.
Use the default docker-compose networking (instead of network_mode: bridge) by removing that line in the compose file.
Assign the localstack service the docker network alias localhost.localstack.cloud.
Let LocalStack know that it has this hostname by setting the environment variable LOCALSTACK_HOSTNAME=localhost.localstack.cloud.

This way you would end up with a docker-compose.yml like this:

version: "3.8"

services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
    image: localstack/localstack
    ports:
      - "127.0.0.1:4510-4559:4510-4559"  # external service port range
      - "127.0.0.1:4566:4566"            # LocalStack Edge Proxy
    environment:
      - SERVICES=${SERVICES-}
      - DEBUG=${DEBUG-}
      - DATA_DIR=${DATA_DIR-}
      - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR-}
      - LOCALSTACK_HOSTNAME=localhost.localstack.cloud
      - OPENSEARCH_ENDPOINT_STRATEGY=path
      - HOST_TMP_FOLDER=${TMPDIR:-/tmp/}localstack
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "${TMPDIR:-/tmp}/localstack:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
    networks:
      default:
        aliases:
          - localhost.localstack.cloud

EDIT: Added endpoint strategy to docker-compose example (see comment below).

ByteProgramming · 2022-03-22T18:34:58Z

@alexrashed

Putting label as "available-in-pro" means Opensearch is not for community-edition?
I tried the steps mentioned above.
Still same issue with endpoint : http://my-demo.eu-central-1.opensearch.localhost.localstack.cloud:4566

I can see in logs for domain creation:
registering an endpoint proxy for http://my-demo.eu-central-1.opensearch.localhost.localstack.cloud:4566 => http://127.0.0.1:40957

Moreover, i see these as well:

localstack_main | 2022-03-22T16:55:29.900:INFO:localstack.services.opensearch.cluster: cluster on http://127.0.0.1:40957 is ready
localstack_main | 2022-03-22T16:55:30.295:DEBUG:localstack.services.opensearch.provider: cluster state polling for my-demo returned! status = True

So cluster is runnning fine.
I am using "
Spring Data Elasticsearch: 4.2.3 , Elasticsearch Client used: 7.12.1, Latest localstack release.

For OPENSEARCH_ENDPOINT_STRATEGY=path, endpoint should be same or different?

Error:

Caused by: java.lang.RuntimeException: Connection refused
2022-03-22T17:03:28.082000+00:00 2022/03/22/[LATEST]0be424a6 	at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.translateException(ElasticsearchRestTemplate.java:400) ~[task/:na]
2022-03-22T17:03:28.082000+00:00 2022/03/22/[LATEST]0be424a6 	... 83 common frames omitted

alexrashed · 2022-03-23T08:20:46Z

Hi again!

No, it means that the DNS resolution using the LocalStack DNS in Lambdas is only available in Pro.
As stated in the documentation, and represented in the result of the create-domain command, the endpoint is different when you change the endpoint strategy. You can find a concise documentation on how the different endpoints look like for the different strategies in our docs.

Unfortunately I forgot to add the endpoint strategy in the docker-compse example above. I edited the example in my previous comment.
When the endpoint strategy is set to path, the endpoint has to look different (but it will be shown correctly in the response of the create-domain command).

ByteProgramming · 2022-03-24T02:13:05Z

HI,
Endpoint : http://localhost.localstack.cloud:4566/opensearch/eu-central-1/my-demo
This URL structure : http://localhost:port/.../.. is not supported by the RestHighLevelClient.
Moreover, i gave a try with extracted host and port from the url like this:
host: localhost.locastack.cloud
port: 4566
So, Created URL : http://localhost.localstack.cloud:4566 and pass this as endpoint to RestHighLevelClient
and got this issue:
Unable to parse response body; nested exception is ElasticsearchStatusException[Unable to parse response body]; nested: ResponseException[method [POST], host [http://localhost.localstack.cloud:4566], URI [/_bulk?timeout=1m], status line [HTTP/1.1 404 ]

Meanwhile, i will try by changing the client to use the original endpoint but i would like to know: if this is the only way it can be resolved for community edition?

alexrashed · 2022-03-24T07:31:46Z

Endpoint : http://localhost.localstack.cloud:4566/opensearch/eu-central-1/my-demo
This URL structure : http://localhost:port/.../.. is not supported by the RestHighLevelClient.

By that you mean that your client does not support a URL path?

So, Created URL : http://localhost.localstack.cloud:4566/ and pass this as endpoint to RestHighLevelClient

Unfortunately this won't be possible because the endpoint strategy path needs the URL path.

Meanwhile, i will try by changing the client to use the original endpoint but i would like to know: if this is the only way it can be resolved for community edition?

No, there's something else you can try. The next option would be to use the endpoint strategy port, as described in our documentation.
In this case you need to make sure to expose the external service port range (which is done in my docker-compose example above).

ByteProgramming · 2022-03-24T08:27:38Z

@alexrashed
yes, RestHighLevelClient for elasticsearch does not support the URL structure.
It supports structure like this only http://localhost:port.
Moreover this client is provided by elastisearch itself. [import org.elasticsearch.client.RestHighLevelClient;]

As given in doc, for index creation we should try like this:
curl -X PUT my-domain.us-east-1.es.localhost.localstack.cloud:4566/my-index

As i have pointed now to "port", i have endpoint as "http://localhost:4510",
so, i fired for index creation as
curl -X PUT localhost:4510/my-index

I got issue:
curl: (56) Recv failure: Connection reset by peer

is creation of index i am doing it correctly?

alexrashed · 2022-03-24T09:31:45Z

Actually, it seems like the RestHighLevelClient from ElasticSearch does support setting a URL path prefix:
https://discuss.elastic.co/t/resthighlevelclient-accessing-an-elastic-http-endpoint-behind-reverse-proxy/117306/4

Please be aware that the latest versions of the elasticsearch clients might not work with OpenSearch (Elastic and AWS had a little dispute there). I don't know if that's the case for the client you are using though.

alexrashed · 2022-03-28T09:42:27Z

Looks like you've been able to resolve your issues. I'm closing this one for now.

ByteProgramming · 2022-03-30T10:54:30Z

No. I am not able to communicate with the approaches given.

alexrashed · 2022-03-30T11:02:11Z

Could you please provide some more information?
So you are using the endpoint strategy path and configure your Lambda such that it uses the path correctly (as described in the link above)?

How does your setup now look like exactly? Could you please share your docker-compose.yaml?
Could you please share the logs of the Lambda and LocalStack when the issue arises?

ByteProgramming · 2022-03-30T13:46:04Z

HI @alexrashed , providing all information:

Started localstack by docker-compose up.

Below is Docker-compose.yml

version: "3.3"

services:
  localstack:
    container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
    image: localstack/localstack
    ports:
      - "127.0.0.1:53:53"                # only required for Pro (DNS)
      - "127.0.0.1:53:53/udp"            # only required for Pro (DNS)
      - "127.0.0.1:443:443"              # only required for Pro (LocalStack HTTPS Edge Proxy)
      - "127.0.0.1:4510-4559:4510-4559"  # external service port range
      - "127.0.0.1:4566:4566"            # LocalStack Edge Proxy
    environment:
      - DEBUG=1
      - DATA_DIR=${DATA_DIR-}
      - LAMBDA_EXECUTOR=${LAMBDA_EXECUTOR-}
      - LOCALSTACK_API_KEY=${LOCALSTACK_API_KEY-}  # only required for Pro
      - LOCALSTACK_HOSTNAME=localhost.localstack.cloud
      - OPENSEARCH_ENDPOINT_STRATEGY=path
      - HOST_TMP_FOLDER=${TMPDIR:-/tmp/}localstack
      - DOCKER_HOST=unix:///var/run/docker.sock
    volumes:
      - "${TMPDIR:-/tmp}/localstack:/tmp/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"
    networks:
      default:
        aliases:
          - localhost.localstack.cloud

Created a domain in Opensearch
aws opensearch create-domain --domain-name dm-demo --endpoint-url http://localhost:4566
```
Result of the domain creation:
```

  {
   "DomainStatus": {
       "DomainId": "000000000000/dm-demo",
       "DomainName": "dm-demo",
       "ARN": "arn:aws:es:eu-central-1:000000000000:domain/dm-demo",
       "Created": true,
       "Deleted": false,
       "Endpoint": "localhost.localstack.cloud:4566/opensearch/eu-central-1/dm-demo",
       "Processing": true,
       "UpgradeProcessing": false,
       "EngineVersion": "OpenSearch_1.1",
       "ClusterConfig": {
           "InstanceType": "m3.medium.search",
           "InstanceCount": 1,
           "DedicatedMasterEnabled": true,
           "ZoneAwarenessEnabled": false,
           "DedicatedMasterType": "m3.medium.search",
           "DedicatedMasterCount": 1,
           "WarmEnabled": false,
           "ColdStorageOptions": {
               "Enabled": false
           }
       },
       "EBSOptions": {
           "EBSEnabled": true,
           "VolumeType": "gp2",
           "VolumeSize": 10,
           "Iops": 0
       },
       "AccessPolicies": "",
       "SnapshotOptions": {
           "AutomatedSnapshotStartHour": 0
       },
       "CognitoOptions": {
           "Enabled": false
       },
       "EncryptionAtRestOptions": {
           "Enabled": false
       },
       "NodeToNodeEncryptionOptions": {
           "Enabled": false
       },
       "AdvancedOptions": {
           "override_main_response_version": "false",
           "rest.action.multi.allow_explicit_index": "true"
   		},
       "ServiceSoftwareOptions": {
           "CurrentVersion": "",
           "NewVersion": "",
           "UpdateAvailable": false,
           "Cancellable": false,
           "UpdateStatus": "COMPLETED",
           "Description": "There is no software update available for this domain.",
           "AutomatedUpdateDate": "1969-12-31T16:00:00-08:00",
           "OptionalDeployment": true
       },
       "DomainEndpointOptions": {
           "EnforceHTTPS": false,
           "TLSSecurityPolicy": "Policy-Min-TLS-1-0-2019-07",
           "CustomEndpointEnabled": false
       },
       "AdvancedSecurityOptions": {
           "Enabled": false,
           "InternalUserDatabaseEnabled": false
       },
       "AutoTuneOptions": {
           "State": "ENABLE_IN_PROGRESS"
       }
   }
}

I can see proxy registration happening:
LOGS:

localstack_main | 2022-03-30T12:51:54.971:INFO:localstack.services.opensearch.cluster: registering an endpoint proxy for http://localhost.localstack.cloud:4566/opensearch/eu-central-1/dm-demo => http://127.0.0.1:37651

localstack_main | 2022-03-30T12:52:18.048:INFO:localstack.services.opensearch.cluster: [37651] [2022-03-30T12:52:18,048][INFO ][o.o.h.AbstractHttpServerTransport] [2b50ca452773] publish_address {127.0.0.1:37651}, bound_addresses {127.0.0.1:37651}
localstack_main | 2022-03-30T12:52:18.049:INFO:localstack.services.opensearch.cluster: [37651] [2022-03-30T12:52:18,049][INFO ][o.o.n.Node ] [2b50ca452773] started

localstack_main | 2022-03-30T12:52:18.194:INFO:localstack.services.opensearch.cluster: cluster on http://127.0.0.1:37651 is ready
localstack_main | 2022-03-30T12:52:18.437:DEBUG:localstack.services.opensearch.provider: cluster state polling for dm-demo returned! status = True

Created an index
curl -X PUT http://localhost.localstack.cloud:4566/opensearch/eu-central-1/dm-demo/my-index

Result:
{"acknowledged":true,"shards_acknowledged":true,"index":"my-index"}

LOGS:
localstack_main | 2022-03-30T13:02:22.456:INFO:localstack.services.opensearch.cluster: [37651] [2022-03-30T13:02:22,455][INFO ][o.o.c.m.MetadataCreateIndexService] [2b50ca452773] [my-index] creating index, cause [api], templates [], shards [1]/[1]
Started other localstack services lambda, dynamodb, s3
Configured my RestHighLevelClient with setPathPrefix as below:

Endpoint passed is : "http://localhost.localstack.cloud:4566"

@Override
public RestHighLevelClient elasticsearchClient() {
  return new RestHighLevelClient(RestClient.builder(HttpHost.create(endpoint)).setPathPrefix("/opensearch/eu-central-1/dm-demo"));
}

Now, with this "path" approach, when i am trying to save data to index(my-index), it giving me error:

022-03-30T13:37:19.640000+00:00 2022/03/30/[LATEST]eb68bb52 Caused by: org.elasticsearch.client.ResponseException: method [PUT], host [http://localhost.localstack.cloud:4566], URI [/opensearch/eu-central-1/dm-demo/my-index?master_timeout=30s&timeout=30s], status line [HTTP/1.1 404 ]
2022-03-30T13:37:19.640000+00:00 2022/03/30/[LATEST]eb68bb52 <?xml version="1.0" encoding="UTF-8"?>
2022-03-30T13:37:19.641000+00:00 2022/03/30/[LATEST]eb68bb52 <Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><BucketName>opensearch</BucketName><RequestID>7a62c49f-347e-4fc4-9331-6e8eEXAMPLE</RequestID></Error>

alexrashed · 2022-03-30T14:17:59Z

Seems like you have a typo in your example:

You create the domain my-demo.
This results in the endpoint http://localhost.localstack.cloud:4566/opensearch/eu-central-1/my-demo/ for the domain.
The path prefix you set in your RestHighLevelClient is .../dm-demo, not .../my-demo.

Does it work if you change the path prefix to the new domain name?

ByteProgramming · 2022-03-30T19:44:11Z

HI @alexrashed
The domain created is "dm-demo" only. It is visible in logs.
Corrected the typo for index command.
Domain : dm-demo
index: my-index

ByteProgramming · 2022-03-31T14:51:05Z

#update.
It seems Localstack elasticsearch is not compatible with rest client from spring-boot-starter-data-elasticsearch.
I tried to use same client from org.elasticsearch.client. It worked.

alexrashed · 2022-03-31T15:17:02Z

I'm happy it works for you now.

But I just need to stress that there is no such thing as a "LocalStack ElasticSearch". With your commands you actually started an OpenSearch cluster. As mentioned above, AWS and Elastic have a troublesome history and lots of clients which are developed by Elastic are explicitly not compatible with OpenSearch (on purpose).

If you want to explicitly start an ElasticSearch cluster, you can so so as well. Here's the docs on our ElasticSearch integration. However, AWS - and therefore LocalStack - will not support any newer ElasticSearch versions than 7.10 (due to their changed license).

ivangfr · 2022-06-14T05:18:43Z

Hi @alexrashed,

It looks like you are the expert here! That is why I am pinging you. My question is related a bit to this issue. That is why I am writing here. So...

I've configured OpenSearch in LocalStack using OPENSEARCH_ENDPOINT_STRATEGY=path

Everything works fine and my OpenSearch URL is http://localhost:4566/opensearch/eu-west-1/my-domain

I have a Spring Boot app that connects perfectly to it when running the app with Maven command.

Now, I've dockerized the app.

I am running the app Docker container in the same network as LocalStack Docker container.

Let's say I've named the LocalStack Docker container as localstack.

Once inside app Docker container, shouldn't we be able to access OpenSearch through the following URL?

http://localstack:4566/opensearch/eu-west-1/my-domain

Unfortunately, it's not working.

For instance, in the host machine, when calling curl http://localhost:4566/opensearch/eu-west-1/my-domain, I get

{
  "name" : "42eb757e08ad",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "dfs1AzvNQsiDkjKri8cIHA",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.1.0",
    "build_type" : "tar",
    "build_hash" : "15e9f137622d878b79103df8f82d78d782b686a1",
    "build_date" : "2021-10-04T21:29:03.079792Z",
    "build_snapshot" : false,
    "lucene_version" : "8.9.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Inside app Docker container, when calling curl http://localstack:4566/opensearch/eu-west-1/my-domain, I get

{}

Thanks!

alexrashed · 2022-06-14T06:51:04Z

Hi @ivangfr!
Please follow the guide in my post above (#5723 (comment)).
You need to set additional configuration parameters for LocalStack to be able to route these requests, coming from an external container, to OpenSearch.
If this doesn't work for you, please open a new issue with the label "question" (comments on closed issues might be overlooked, and it's dangerous to mix different issues).

ivangfr · 2022-06-14T09:14:36Z

Thanks @alexrashed! It worked! Sorry for commenting on a closed issue.

yoav-klein · 2023-03-14T10:52:38Z

@alexrashed I have a related question if you may:

I want to run OpenSearch in LocalStack which I want to run in a Docker container.
Now, I want to have a fluentd container to send data to the OpenSearch cluster.

I don't think I can use the domain endpoint strategy since the fluentd container won't resolve
the domain name correctly, so I'm trying for the port strategy.

Overall, I have the following docker-compose file:

version: "3.9"
services:
 fluentd:
    image: quay.io/fluentd_elasticsearch/fluentd:v3.1.0
    hostname: fluentd
    volumes:
      - type: bind
        source: ./
        target: /etc/fluent/config.d
    ports:
        - "9880:9880"
 localstack :
    image: localstack/localstack
    environment:
      OPENSEARCH_ENDPOINT_STRATEGY: port
#      EXTERNAL_SERVICE_PORT_START: 4515
#     EXTERNAL_SERVICE_PORT_END: 4516
    ports:
      - "4566:4566"
      - "4510-4559:4510-4559" # optional - for inbound agents

Running this, and then running inside the localstack container:

$ awslocal opensearch create-domain --domain-name=my-domain

This will give me the endpoint for my domain:

{
    "DomainStatus": {
        "DomainId": "000000000000/my-domain",
        "DomainName": "my-domain",
        "ARN": "arn:aws:es:us-east-1:000000000000:domain/my-domain",
        "Created": true,
        "Deleted": false,
        "Endpoint": "localhost:4510",

Now, I've configured my fluentd as such:

# fluentd.conf
<source>
  @type http
  port 9880
  bind "0.0.0.0"
</source>
<match **>
 @type elasticsearch
  host localstack
  port 4510
  index_name fluentd
  type_name fluentd
</match>

My questions are:

Can I be sure what port will be allocated for the cluster so that I can configure it in the fluentd.conf file? I've tried using the EXTERNAL_SERVICE_PORT_START and END variables with no luck.
Is this the best approach for doing this?

alexrashed · 2023-03-15T09:08:10Z

Hi @yoav-klein!
It seems the port strategy isn't the best pick for you, because you won't know which port LocalStack is going to select for the OpenSearch domain. You could try using the path strategy or configure the fluentd container after the domain has been created (and you know the endpoint / how to reach it).
However, please create a new issue if you are experiencing any further issues. Closed issues are hard to track and different topics get intermingled quite easily.

yoav-klein · 2023-03-17T13:50:09Z

Hi @alexrashed, I think the path strategy is the right way for me. Since we're already here just let me ask you: In your comment above you specified the following actions:

Use the default docker-compose networking (instead of network_mode: bridge) by removing that line in the compose file.
Assign the localstack service the docker network alias localhost.localstack.cloud.
Let LocalStack know that it has this hostname by setting the environment variable LOCALSTACK_HOSTNAME=localhost.localstack.cloud.

Can you explain why these steps are required? Seemingly, I can just run localstack as a container in the docker-compose.yaml file, and to access it using the container name from another container in the same docker-compose.

alexrashed · 2023-03-20T07:23:27Z

Hi @yoav-klein.
This was a specific response for the requested use case. But here's the reasoning:

Network mode bridge: This changes the networking for LocalStack in Docker-Compose. New Lambdas are spawned as separate docker containers, which should run in the same docker network as LocalStack (to be able to access it and use Docker's DNS to resolve the LocalStack container).
The docker network alias and LOCALSTACK_HOSTNAME are used to:
- Tell LocalStack how it can be reached (by setting the LOCALSTACK_HOSTNAME). This is necessary for when LocalStack spawns Lambdas or generates URLs.
- Actually configure the Docker network to resolve localhost.localstack.cloud to LocalStack.

In my explanations I chose localhost.localstack.cloud because this host can also be resolved to localhost by your host machine (in case you want to connect to LocalStack from your host machine as well with the same URLs generated by LocalStack).

theahura · 2023-09-12T19:17:08Z

FYI @alexrashed stumbled on this issue from google -- noticed all of the links to documentation that you provided now 404. Would be fantastic if those could be updated!

For posterity, I think the link is: https://docs.localstack.cloud/user-guide/aws/opensearch/

ByteProgramming added the type: question Please ask questions on discuss.localstack.cloud label Mar 22, 2022

whummer added the aws:opensearch Amazon OpenSearch Service label Mar 22, 2022

alexrashed self-assigned this Mar 22, 2022

alexrashed added the available-in-pro label Mar 22, 2022

alexrashed closed this as completed Mar 28, 2022

alexrashed reopened this Mar 30, 2022

alexrashed closed this as completed Mar 31, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to Connect to localstack's OpenSearch with the Endpoint provided #5723

Unable to Connect to localstack's OpenSearch with the Endpoint provided #5723

ByteProgramming commented Mar 22, 2022 •

edited

github-actions bot commented Mar 22, 2022

alexrashed commented Mar 22, 2022

ByteProgramming commented Mar 22, 2022

alexrashed commented Mar 22, 2022 •

edited

ByteProgramming commented Mar 22, 2022 •

edited

alexrashed commented Mar 23, 2022

ByteProgramming commented Mar 24, 2022 •

edited

alexrashed commented Mar 24, 2022 •

edited

ByteProgramming commented Mar 24, 2022 •

edited

alexrashed commented Mar 24, 2022

alexrashed commented Mar 28, 2022

ByteProgramming commented Mar 30, 2022

alexrashed commented Mar 30, 2022

ByteProgramming commented Mar 30, 2022 •

edited

alexrashed commented Mar 30, 2022

ByteProgramming commented Mar 30, 2022

ByteProgramming commented Mar 31, 2022

alexrashed commented Mar 31, 2022

ivangfr commented Jun 14, 2022

alexrashed commented Jun 14, 2022

ivangfr commented Jun 14, 2022

yoav-klein commented Mar 14, 2023

alexrashed commented Mar 15, 2023

yoav-klein commented Mar 17, 2023

alexrashed commented Mar 20, 2023

theahura commented Sep 12, 2023 •

edited

Unable to Connect to localstack's OpenSearch with the Endpoint provided #5723

Unable to Connect to localstack's OpenSearch with the Endpoint provided #5723

Comments

ByteProgramming commented Mar 22, 2022 • edited

Is there an existing issue for this?

Question

Anything else?

github-actions bot commented Mar 22, 2022

alexrashed commented Mar 22, 2022

ByteProgramming commented Mar 22, 2022

alexrashed commented Mar 22, 2022 • edited

ByteProgramming commented Mar 22, 2022 • edited

alexrashed commented Mar 23, 2022

ByteProgramming commented Mar 24, 2022 • edited

alexrashed commented Mar 24, 2022 • edited

ByteProgramming commented Mar 24, 2022 • edited

alexrashed commented Mar 24, 2022

alexrashed commented Mar 28, 2022

ByteProgramming commented Mar 30, 2022

alexrashed commented Mar 30, 2022

ByteProgramming commented Mar 30, 2022 • edited

alexrashed commented Mar 30, 2022

ByteProgramming commented Mar 30, 2022

ByteProgramming commented Mar 31, 2022

alexrashed commented Mar 31, 2022

ivangfr commented Jun 14, 2022

alexrashed commented Jun 14, 2022

ivangfr commented Jun 14, 2022

yoav-klein commented Mar 14, 2023

alexrashed commented Mar 15, 2023

yoav-klein commented Mar 17, 2023

alexrashed commented Mar 20, 2023

theahura commented Sep 12, 2023 • edited

ByteProgramming commented Mar 22, 2022 •

edited

alexrashed commented Mar 22, 2022 •

edited

ByteProgramming commented Mar 22, 2022 •

edited

ByteProgramming commented Mar 24, 2022 •

edited

alexrashed commented Mar 24, 2022 •

edited

ByteProgramming commented Mar 24, 2022 •

edited

ByteProgramming commented Mar 30, 2022 •

edited

theahura commented Sep 12, 2023 •

edited