Can't put document into AWS ES service.

[mthenw]

Which version of Elastic are you using?

elastic.v2 (for Elasticsearch 1.x)

Please describe the expected behavior

Successful document put into the index.

Please describe the actual behavior

Error is returned:

elastic: Error 403 (Forbidden)

Any steps to reproduce the behavior?

Setup:

    creds := credentials.NewEnvCredentials()
    signer := v4.NewSigner(creds)
    awsClient, err := aws_signing_client.New(signer, nil, "es", "us-west-2")
    if err != nil {
        return nil, err
    }

    return elastic.NewClient(
        elastic.SetURL(...),
        elastic.SetScheme("https"),
        elastic.SetHttpClient(awsClient),
        elastic.SetSniff(false),
    )

Put:

    _, err = e.Client.Index().Index(indexName).Type(indexType).
        Id(doc.ID).
        BodyJson(doc).
        Do()

Not sure if this is elastic or aws_signing_client issue.

[olivere]

@sha1sum Hi Anthony, do you have the time to reproduce this issue?

[olivere]

#301 and #303 use a different method for signing. If you're in a hurry, you could try that.

[mthenw]

I've solved that by creating http.Transport based on https://github.com/smartystreets/go-aws-auth

type AWSSigningTransport struct {
    HTTPClient  *http.Client
    Credentials awsauth.Credentials
}

// RoundTrip implementation
func (a AWSSigningTransport) RoundTrip(req *http.Request) (*http.Response, error) {
    return a.HTTPClient.Do(awsauth.Sign4(req, a.Credentials))
}

Usage

    signingTransport := AWSSigningTransport{
        Credentials: awsauth.Credentials{
            AccessKeyID:     os.Getenv("AWS_ACCESS_KEY"),
            SecretAccessKey: os.Getenv("AWS_SECRET_KEY"),
        },
        HTTPClient: http.DefaultClient,
    }
    signingClient := &http.Client{Transport: http.RoundTripper(signingTransport)}

    return elastic.NewClient(
        elastic.SetURL(...),
        elastic.SetScheme("https"),
        elastic.SetHttpClient(signingClient),
        elastic.SetSniff(false),
    )

[mthenw]

@olivere I found another bug. Looks like elastic creates wrong URL path (https://github.com/olivere/elastic/blob/release-branch.v2/search.go#L311-L344).

Requests go to URL//_search?search_type=count (double slash) instead of URL/_search?search_type=count. So for all search queries request signing doesn't work because of signature mismatch.

[olivere]

Yep. That's a bug. Thanks. Workaround is to use _all as an index name. Will hopefully find some time on the weekend.

[olivere]

The issue with the wrong URL being generated when indices and types are missing should be fixed in the next release later today.

I'm closing this issue for now as the problem with signing seems to be related to aws_signing_client#1, not elastic.

Feel free to re-open if this doesn't solve your problem. Thanks for your support.

[AlmogBaku]

I'm still having the same issue

[olivere]

@AlmogBaku If you've found a problem, don't be afraid to file a new issue. Put in a back-reference to this issue if you like. This issue actually talks about two bugs, and I'm not sure which one you're refering to.

[AlmogBaku]

actually, it was a bug of mine.. the included illegal chars which probably caused the failure to sign it. sorry for the trouble

[Single430]

This is the problem I encountered， help me!

no Elasticsearch node available
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x636df6]

[olivere]

@Single430 Can you post the code please? The 2nd line is because you probably didn't do any error checks and tried to use the *Client although it wasn't property initialized and is nil.

[Single430]

@olivere Ｏk , The first time use, there are mistakes please point out, thanks

type AWSSigningTransport struct {
    HTTPClient  *http.Client
    Credentials awsauth.Credentials
}

func (a AWSSigningTransport) RoundTrip(req *http.Request) (*http.Response, error) {
    return a.HTTPClient.Do(awsauth.Sign4(req, a.Credentials))
}

func EsSearch() {
    fmt.Println("search...")
    signingTransport := AWSSigningTransport{
        Credentials: awsauth.Credentials{
            AccessKeyID:     "xxxxxxxx",
            SecretAccessKey: "xxxxxxxxxxxxx",
        },
        HTTPClient: http.DefaultClient,
    }
    httpClient := &http.Client{
        Transport: http.RoundTripper(signingTransport),
    }
    client, err := elastic.NewClient(
        elastic.SetURL("https://search-xxxxxxx-xxxxxxx.cn-northwest-1.es.amazonaws.com.cn"),
        //elastic.SetScheme("https"),
        elastic.SetHealthcheckInterval(10*time.Second),
        elastic.SetHttpClient(httpClient),
        elastic.SetSniff(false),
        )
    if err != nil {
        fmt.Println(err)
    }
    
    termQuery := elastic.NewRangeQuery("published_time").Gte("2018-03-20T12:00:00").Lte("2018-03-21T12:00:00")
    searchResult, err := client.Search().
        Index("index").Type("type").
        Query(termQuery).
        Sort("published_time", true).
        From(0).Size(10).
        Pretty(true).
        Do()
    if err != nil {
        panic(err)
    }
    
    fmt.Printf("Query took %d milliseconds\n", searchResult.TookInMillis)
    fmt.Println(searchResult.Hits.TotalHits)
    
    fmt.Println("Hello World!")
}

func main() {
    EsSearch()
}

[olivere]

@Single430 I will setup a cluster on AWS myself for testing. It will take a few minutes.

[olivere]

@Single430 I just successfully tested this snippet of code against an Elasticsearch Service 6.2 cluster on AWS.

$ go run main.go -url=https://search-xxxx-yyyy.eu-central-1.es.amazonaws.com
Connection succeeded

It must be something in your code.

[olivere]

@Single430 Recipe is here.

[Single430]

@olivere I used your code, local is ok, but aws is no

go run main.go -access-key=xxxxx -secret-key=xxxxxxxx

Connection succeeded

-----------------------

go run main.go -access-key=xxxxx -secret-key=xxxxxxxx -url=https://search-xxxxxx-xxxxxxxxx.es.amazonaws.com.cn

no Elasticsearch node available
exit status 1

[olivere]

@Single430 As it worked at my side, it can only be something in your code or with AWS. Maybe you are not allowed to connect to your cluster?

[Single430]

python is ok, use golang is no, access-key, secret-key, url all are correct......................................

thanks very much!!!! @olivere

[Single430]

I do not give up, in the end there is a problem there @olivere

[olivere]

@Single430 Well, I don't know what else I could do. Let me know when you finally found the underlying core issue.

[Single430]

of course @olivere

[Single430]

What is the version of elasticsearch on your aws ？ @olivere

[olivere]

@Single430 I used the current version of Elastic (6.1.12) with AWS Elasticsearch Service 6.2.

[Single430]

@olivere The problem was found. I was using the go-aws-auth for verification but found this to be a problem with the package.

my region is cn-northwest-1, service is es, but use go-aws-auth only region (us-east-1), service (s3), and my endpoint is https://search-xxxx-xxxxxxxxxxxxxxx.cn-northwest-1.es.amazonaws.com.cn/ ,

result is modify go-aws-auth package in common.go - > func serviceAndRegion()

[olivere]

@Single430 Great. You fixed it :-)

If you have the time to dig deeper: AWS has its own official Go library that includes a signer as well (see https://docs.aws.amazon.com/sdk-for-go/api/aws/signer/v4/). And I found this (old) entry in the Wiki, which illustrates how to do it with the official AWS library. Maybe that's a cleaner solution for you.

[Single430]

@olivere Thanks very mush!!! Ｉ also solved the problem with the method you provided, I really appreciate it.