Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] enhance awscurl support create/send multipart POST REST request (from file) #194

Closed
ghost opened this issue Mar 8, 2024 · 6 comments
Closed

[FEATURE] enhance awscurl support create/send multipart POST REST request (from file) #194

ghost opened this issue Mar 8, 2024 · 6 comments
Assignees
@okigan
Labels
enhancement

Comments

@ghost
Copy link

ghost commented Mar 8, 2024

Versions

  • OS :
    • cURL => via Windows 10
    • awscurl => via gitlab runner (Docker)
  • OpenSearch 2.11

What's my problem ?
First of all, this closed issue : opensearch-project/opensearch-cli#76
Seems to be close to my problem, but I'm not using opensearch-cli curl. Only awscurl

I try to migrate dashboards objects from one opensearch cluster to another one.
To do that I'm using api provided by opensearch : /_dashboards/api/saved_objects/_import?overwrite=true
But I get errors when I'm trying to post my objects.

How that's done ?
I'm exporting the objects via the opensearch UI (From my first cluster). Dashboards Management > Saved objects > Export (when my dashboard is selected). And I get my objects in a .ndjson file.
After that i'm posting my .ndjson file with awscurl :

Without Content Type header

awscurl --service es --region ${region} -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
  -H 'osd-xsrf: true' \
  -d '${dashboard_content}'
{
   "statusCode":415,
   "error":"Unsupported Media Type",
   "message":"Unsupported Media Type"
}

Without Boundary

awscurl --service es --region ${region} -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
  -H 'Content-Type: multipart/form-data' \
  -H 'osd-xsrf: true' \
  -d '${dashboard_content}'
{
   "statusCode":400,
   "error":"Bad Request",
   "message":"Invalid content-type header: multipart missing boundary"
}

With a random boundary

awscurl --service es --region ${region} -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
  -H 'Content-Type: multipart/form-data; boundary=---------BOUNDARY;' \
  -H 'osd-xsrf: true' \
  -d '${dashboard_content}'

image

image

By only using curl, and don't put Content Type header, it works fine.

curl -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
  -H "osd-xsrf: true"
  -F "file=@data/test_dashboard.ndjson"
{
   "successCount":6,
   "success":true,
   "successResults":[
      {
         "type":"index-pattern",
         "id":"610267e0-7f21-11ee-80e6-370892a64263",
         "meta":{
            "title":"documents-metadata-v2",
            "icon":"indexPatternApp"
         },
         "overwrite":true
      },
      {
         "type":"visualization",
         "id":"da95cc90-7f22-11ee-9e31-e14f5ec8461b",
         "meta":{
            "title":"count by cat_doc",
            "icon":"visualizeApp"
         },
         "overwrite":true
      },
      {
         "type":"index-pattern",
         "id":"d3d7af60-4c81-11e8-b3d7-01146121b73d",
         "meta":{
            "title":"opensearch_dashboards_sample_data_flights",
            "icon":"indexPatternApp"
         },
         "overwrite":true
      },
      {
         "type":"search",
         "id":"571aaf70-4c88-11e8-b3d7-01146121b73d",
         "meta":{
            "title":"[Flights] Flight Log",
            "icon":"discoverApp"
         },
         "overwrite":true
      },
      {
         "type":"visualization",
         "id":"94003870-dbb5-11ee-81ba-b7ae96977aa8",
         "meta":{
            "title":"test_object_dashboard",
            "icon":"visualizeApp"
         },
         "overwrite":true
      },
      {
         "type":"dashboard",
         "id":"2d7aceb0-7f23-11ee-9e31-e14f5ec8461b",
         "meta":{
            "title":"test_dashboard",
            "icon":"dashboardApp"
         },
         "overwrite":true
      }
   ]
}
@okigan
Copy link
Owner

okigan commented Mar 8, 2024 via email

@ghost
Copy link
Author

ghost commented Mar 12, 2024

Yes, cURL doesn't need to explicit the Content Type.
Here's the cURL with verbose option :

curl -v -X POST ${domain_url}/_dashboards/api/saved_objects/_import?overwrite=true \
-H "osd-xsrf: true" \
-F "file=@data/test_dashboard.ndjson"

And the result :

Note: Unnecessary use of -X or --request, POST is already inferred.
* Uses proxy env variable no_proxy == '****'
*   Trying [::1]:8080...
*   Trying ${domain_ip}:8080...
* Connected to localhost (127.0.0.1) port 8080
> POST /_dashboards/api/saved_objects/_import?overwrite=true HTTP/1.1
> Host: ${domain_url}
> User-Agent: curl/8.4.0
> Accept: */*
> osd-xsrf: true
> Content-Length: 40384
> Content-Type: multipart/form-data; boundary=------------------------wOWPtrGc1TrXmDPpX5GfNt
>
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< Cache-Control: private, no-cache, no-store, must-revalidate
< Content-Length: 947
< Content-Type: application/json; charset=utf-8
< Date: Tue, 12 Mar 2024 09:44:45 GMT
< Osd-Name: osd
< Set-Cookie: security_authentication_saml1=Fe26.2**838c01d554bcb2b7d4e5ead20dafc498fcdd74e1faf6b74c5d1348a2326b713c*D_5paBXbChAFcjTpZ00nLg*YUmGokrp_eV_EEv_2S0NZecOyi1Lwhrgas5VLH69YDC6nBJDau20hsaz8ipoTcUPxMS2AxQoGXqv5bUMo8sY8ulz783TWCck7iibWpBdYn_FvqXpBoCu9C23MAD-v6CDe7CJ5ch2KczWBZhnI-jc_z53Lz-Dt62h2PQ1OAAq7-He5QR7Jx-lg87XpaWpCL2A0-uAvfASBUoZ7ui2IvPHuLmw5myZ5acmFZsxpgg9QLrRKlBlUvpNRz0-rG6EZY2uI1UoVRnsaYFLUkRKe3YOZYSCsvLWxGDi6XBEUfdLH9bl3WSV1u0eDdAEvt0qViIcp8Njt0_iT1iyQiHR6ju-Xg**8def9b476d52a739f02c24f8ed8783b8585d01177dcd04e624d2ec17b0a17c61*nVfuzR87kXvFQPaesZZZfY1aPWiT87yLITvEMUJOj5s; Secure; HttpOnly; Path=/_dashboards
< Set-Cookie: security_authentication_saml2=; Max-Age=0; Expires=Thu, 01 Jan 1970 00:00:00 GMT; Secure; HttpOnly; Path=/_dashboards
< Set-Cookie: security_authentication_saml3=; Max-Age=0; Expires=Thu, 01 Jan 1970 00:00:00 GMT; Secure; HttpOnly; Path=/_dashboards
< Set-Cookie: security_authentication=Fe26.2**59193956774e2ff6b56087734d72a99abd64c27811b759e7c10e85f995ae4792*CFtdPl74zukFptAGTWbAXw*bHObA_QMNUiaQI8V471pQhS_Z2GRkmQuy4-G6e5ycbyBSGZ_8W6ThTG7f3SlTgtQLD5AmqSa6mGKeZ5j6JJZoY0zTX2T5Zimkr8JWEljMBIlFOK_G-SSNmEtZPVuKHUfbKOVH-4_6WlW6DVVQO7rGYe9dnm615ssNvTuzFQNJ8BTZ5HVXutfXhMrOzipcs1nh4A2K6PaZnbT-RNpS823WT_JVmdfNcrHXj2xWfjYbHQLC2gv5FTcOHAMd0yPCDS0**9e47d862c46786b7011a75ddbde639601e554e9a30ca09366ede38184d642392*vsovYjoNWeD0yiX3QUmj3ryV1V0IJTuejNc3rBMOLP8; Secure; HttpOnly; Path=/_dashboards
< Strict-Transport-Security: max-age=31536000
< X-Amzn-Requestid: a56e2412-d6f8-4119-84ac-9eead1de5731
<
{"successCount":6,"success":true,"successResults":[{"type":"index-pattern","id":"610267e0-7f21-11ee-80e6-370892a64263","meta":{"title":"documents-metadata-v2","icon":"indexPatternApp"},"overwrite":true},{"type":"visualization","id":"da95cc90-7f22-11ee-9e31-e14f5ec8461b","meta":{"title":"count by cat_doc","icon":"visualizeApp"},"overwrite":true},{"type":"index-pattern","id":"d3d7af60-4c81-11e8-b3d7-01146121b73d","meta":{"title":"opensearch_dashboards_sample_data_flights","icon":"indexPatternApp"},"overwrite":true},{"type":"search","id":"571aaf70-4c88-11e8-b3d7-01146121b73d","meta":{"title":"[Flights] Flight Log","icon":"discoverApp"},"overwrite":true},{"type":"visualization","id":"94003870-dbb5-11ee-81ba-b7ae96977aa8","meta":{"title":"test_object_dashboard","icon":"visualizeApp"},"overwrite":true},{"type":"dashboard","id":"2d7aceb0-7f23-11ee-9e31-e14f5ec8461b","meta":{"title":"test_dashboard","icon":"dashboardApp"},"overwrite":true}]}

* Connection #0 to host ${domain}left intact
```^

As we can see, cURL putting itself the Content Type to multipart/form-data and setting the boundary to my data.

@okigan
Copy link
Owner

okigan commented Mar 14, 2024 via email

@okigan okigan changed the title [BUG] awscurl cannot create/send multipart POST REST request (from file) [FEATURE] enhance awscurl support create/send multipart POST REST request (from file) Mar 14, 2024
okigan added a commit that referenced this issue Mar 14, 2024
@okigan
[FEATURE] enhance awscurl support create/send multipart POST REST req…
340e4ae 
…uest (from file)

#194
@okigan
Copy link
Owner

okigan commented Mar 14, 2024 •
edited
Loading

@llasherme try out this branch, his are the relevant steps:

git checkout feature/194
make venv
source venv/bin/activate
./run.sh -v -X POST -F name=@README.md https://httpbin.org/post

once that works try with your ES cluster

@okigan okigan self-assigned this Mar 15, 2024
@okigan
Copy link
Owner

okigan commented Mar 15, 2024 •
edited
Loading

@llasherme interesting, try latest please (de473d8)

@ghost ghost closed this as not planned Won't fix, can't repro, duplicate, stale Mar 18, 2024
@okigan
Copy link
Owner

okigan commented Mar 18, 2024 via email

This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@okigan okigan

Labels
enhancement
Projects
None yet
Milestone
No milestone
Development

When branches are created from issues, their pull requests are automatically linked.

feature/194 okigan/awscurl
1 participant
@okigan