Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support gcs storage #1781

Merged
merged 2 commits into from Jul 5, 2023

Conversation

NiwakaDev
Copy link
Collaborator

@NiwakaDev NiwakaDev commented Jun 15, 2023

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

This PR adds support gcs storage.

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

#1611

@NiwakaDev
Copy link
Collaborator Author

integration_http_gcs_test:

        PASS [   0.044s] tests-integration::main integration_http_gcs_test::test_dashboard_path
        PASS [   5.121s] tests-integration::main integration_http_gcs_test::test_health_api
        PASS [   7.221s] tests-integration::main integration_http_gcs_test::test_prometheus_promql_api
        PASS [   7.280s] tests-integration::main integration_http_gcs_test::test_metrics_api
        PASS [   7.442s] tests-integration::main integration_http_gcs_test::test_scripts_api
        PASS [   7.539s] tests-integration::main integration_http_gcs_test::test_sql_api
        PASS [   9.470s] tests-integration::main integration_http_gcs_test::test_prom_http_api

@NiwakaDev
Copy link
Collaborator Author

NiwakaDev commented Jun 15, 2023

integration_grpc_gcs_test:

Context:
    url: https://storage.googleapis.com/upload/storage/v1/b/~/o?uploadType=media&name=8f7e1a5c-a1ed-42fe-a039-b0be2af27c2c/cluster/dn-0/procedure/2fccdf8d-45fb-4e57-b206-52460fad89ca/0000000002.step
    called: http_util::Client::send_async
    service: gcs
    path: cluster/dn-0/procedure/2fccdf8d-45fb-4e57-b206-52460fad89ca/0000000002.step

Source: error sending request for url (https://storage.googleapis.com/upload/storage/v1/b/~/o?uploadType=media&name=8f7e1a5c-a1ed-42fe-a039-b0be2af27c2c/cluster/dn-0/procedure/2fccdf8d-45fb-4e57-b206-52460fad89ca/0000000002.step): dispatch task is gone: runtime dropped the dispatch task

Caused by:
    0: dispatch task is gone: runtime dropped the dispatch task
    1: runtime dropped the dispatch task
    
test integration_grpc_gcs_test::test_dbname ... FAILED

failures:

failures:

        FAIL [   6.364s] tests-integration::main integration_grpc_gcs_test::test_auto_create_table
        FAIL [   6.735s] tests-integration::main integration_grpc_gcs_test::test_dbname
        FAIL [   6.294s] tests-integration::main integration_grpc_gcs_test::test_insert_and_select
        FAIL [   6.498s] tests-integration::main integration_grpc_gcs_test::test_prom_gateway_query

@NiwakaDev
Copy link
Collaborator Author

Right now the integration_http_gcs_test passes, on the other hand the integration_grpc_gcs_test consistently fails.

@NiwakaDev
Copy link
Collaborator Author

@killme2008

I implemented #1611, but it seems like the integration_grpc_gcs_test consistently fails. I don't know why this error happens.

@killme2008
Copy link
Contributor

@NiwakaDev Let me try it. Thanks a lot. There is a conflict have to be resolved.

@NiwakaDev
Copy link
Collaborator Author

@killme2008

There is a conflict have to be resolved.

I addressed this.

@killme2008
Copy link
Contributor

@NiwakaDev I have to apply the GCS account and token, sorry for the late response. I am looking into it today.

@NiwakaDev
Copy link
Collaborator Author

@killme2008

I'm sorry for the late reply.

I am looking into it today.

Thank you for looking it.

@killme2008
Copy link
Contributor

Looks like it's a concurrent issue between tests. The HTTP client was stopped while other tests were running.

I am not quite sure the OpenDAL GCS backend implementation working properly. I'll look into it.

@NiwakaDev
Copy link
Collaborator Author

The HTTP client was stopped while other tests were running.

Thanks a lot for investigating the cause.

I am not quite sure the OpenDAL GCS backend implementation working properly. I'll look into it.

Ok. I'll also re-check gcs setting.

@killme2008
Copy link
Contributor

@NiwakaDev I think we can merge this PR first, the test concurrent issue can be fixed later.

@killme2008 killme2008 marked this pull request as ready for review July 5, 2023 08:13
Copy link
Contributor

@killme2008 killme2008 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@killme2008 killme2008 added the Doc update required This change requires document update on https://github.com/GreptimeTeam/docs label Jul 5, 2023
@killme2008 killme2008 requested review from WenyXu and v0y4g3r July 5, 2023 11:03
@killme2008 killme2008 merged commit 66e5ed5 into GreptimeTeam:develop Jul 5, 2023
22 checks passed
WenyXu pushed a commit to WenyXu/greptimedb that referenced this pull request Jul 7, 2023
paomian pushed a commit to paomian/greptimedb that referenced this pull request Oct 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Doc update required This change requires document update on https://github.com/GreptimeTeam/docs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants