Presto not getting deployed with type: gcp-types/dataproc-v1:projects.regions.clusters
#546
Comments
Priyankasaggu11929
changed the title
type: gcp-types/dataproc-v1:projects.regions.clusters
|
I am not a Dataproc expert, but let me try to help you. |
|
Taking a quick look at the REST APIs: It looks like to me, the GA API does NOT support |
|
@ocsig Thanks for the reply. Yes, I meant, I think I need your help here. I'm not sure how to create a custom type for |
|
Ok, I am working on it right now. Is this script working for you with the GA type ( except the PRESTO installation)? |
|
Oh, @ocsig sorry for the typo there. That happened while I was pasting the template here and removing the actual values with configurable environment variables. Yes, the script works properly apart from the PRESTO installation. |
|
(NOTE: I am still rinning my test, I may modify this post if it fails.) Creation of a custom Type documentation You don't need to modify your template except the type you are using. I made changes to use defaults because it was easier for testing. Step1: options:
inputMappings:
- fieldName: Authorization
location: HEADER
value: $.concat("Bearer ", $.googleOauth2AccessToken())
methodMatch: .*
collectionOverrides:
- collection: projects.regions.clusters
options:
virtualProperties: |
schema: http://json-schema.org/draft-04/schema#
type: object
properties:
region:
type: string
required:
- region
inputMappings:
- methodMatch: ^(create|update|get|patch|delete)$
location: PATH
fieldName: region
value: >
$.resource.properties.region
- methodMatch: ^setIamPolicy$
location: PATH
fieldName: resource
value: >
$.resource.self.name
- methodMatch: ^(update|get|patch|delete)$
location: PATH
fieldName: clusterName
value: >
$.resource.properties.clusterNameStep 2: gcloud beta deployment-manager type-providers create dataproc-v1beta --api-options-file dataproc-v1beta.type.yaml --descriptor-url='https://dataproc.googleapis.com/$discovery
/rest?version=v1beta2'
Waiting for insert [operation-1584697838999-5a14637c5191b-9deb0532-b1336e26]...done.
Created type_provider [dataproc-v1beta]From this point, your project has a custom type: dpbeta.jinja NOTE: Change 'my-project' {% set clusterName = (env["deployment"] + "-dataproc-cluster") %}
resources:
- name: {{ clusterName }}
type: my-project/dataproc-v1beta:projects.regions.clusters
properties:
region: {{ properties["region"] }}
projectId: {{ env["project"] }}
clusterName: {{ clusterName }}
config:
# configBucket: example-bucket
gceClusterConfig:
zoneUri: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/zones/{{ properties["zone"] }}
tags:
- example-firewall-02
# subnetworkUri: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/regions/{{ properties["zone"] }}/subnetworks/example-subnet-1
# internalIpOnly: true
masterConfig:
numInstances: 1
machineTypeUri: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/zones/{{ properties["zone"] }}/machineTypes/n1-standard-2
diskConfig:
bootDiskSizeGb: 200
bootDiskType: pd-ssd
workerConfig:
numInstances: 2
machineTypeUri: https://www.googleapis.com/compute/v1/projects/{{ env["project"] }}/zones/{{ properties["zone"] }}/machineTypes/n1-standard-2
diskConfig:
bootDiskSizeGb: 200
bootDiskType: pd-ssd
softwareConfig:
imageVersion: 1.4.23-ubuntu18
optionalComponents:
- ANACONDA
- HIVE_WEBHCAT
- JUPYTER
- PRESTO
- ZEPPELIN
- ZOOKEEPER dpbeta.yaml imports:
- path: dpbeta.jinja
resources:
- name: dpbeta
type: dpbeta.jinja
properties:
region: us-west1
zone: us-west1-bCreating the deployment: gcloud deployment-manager deployments create dptest --config=dpbeta.yaml |
|
Thanks for the steps. I'm reproducing the steps and testing now. |
|
@ocsig I followed the steps and I think it happened as well. But now I'm stuck at this error I was reading this https://cloud.google.com/deployment-manager/docs/access-control but still couldn't find out how to provide permission for the same. |
|
Is your type in the same project where you are launching the deployment from? ( so {{service_account_number}} == {{project_name}} ( The Project number and the project ID is identifying the same project, the two value is actually different?) |
|
Make sure you either have the type in every project where you want to use it OR every DM Service account is a |
|
I am using my organisation's gcp platform, so even after I'm (corresponding user account) granted with Editor role, the permissions are still not satisfied. I'll update here once it gets solved Or in otherwise case too. |
|
[UPDATES] I exhaustively added all the roles available for Deployment manager to the required service account.
I re-checked the project name as well to ensure if it is right. Also, I realised we have only project, so no chance of having the custom type-provider in another project. But I keep getting the same error down below.
Nothing seem to resolve the Permission error. |
|
Can you confirm that the reducted projectID is correct? |
|
yes, project ID is correct. I re-checked it. Also, when I'm trying to describe the custom type-provider through gcloud, the selflink comes as something like This
|
|
It is pretty hard to debug your setup like this, so please forgive me for the super basic checks. What I am trying to verify if everything is happening in the same project or at one place your code was pointing to somewhere else. Lets say, the project where you want to use this type is has the ID Would you mind to go throught the following checklist and let me know if you find any deviation?
Please let me know which point(s) are failing in the checklist and what value do you see instead of your ProjectID/Number. ( Is it an other ID of yours you see?) |
|
Ok, giving more clarification.
Now, when I run the above checklists, I get the following outputs.
Output: Here in place of getting
Output: Again, I get
Note: If it doesn't makes proper sense even now, may I drop you an email with proper screenshots or actual information, whatsoever help you the best. Thank you once again. :) |
|
@ocsig I updated the above comment. |
|
Great, I believe I have an understanding of the problem.
The issue is, that the type you would like to use is NOT Would you mind to update the ( And because you do not have listing permision to the project which has a ProjectID='abc', you are getting permission error.) Let me know if this solves your problem. |
|
@ocsig yes, I tried doing the above change sometime back only. It gives me the following error: |
|
The good news is, this is a different error now. Now we know you are using the custom type, because this error actually comes from communicating with the Dataproc API. Can you verify your type configuration contains the authentication part: Running 'gcloud beta deployment-manager type-providers describe dataproc-v1beta' should contain this at the end: [....]
options:
inputMappings:
- fieldName: Authorization
location: HEADER
methodMatch: .*
value: $.concat("Bearer ", $.googleOauth2AccessToken())
[.....]If this is missing, your type did not picked up the config file |
|
Yes, It comes in the output. Ok, I think I found a problem. In place of It is running currently. I will let you know if it worked successfully or not. No errors so far. Now, I feel super funny as all this was because of a singe space typo. :| |
|
Wow, I knew that there has to be a typo, but couln't spot it. From this point every property should be properly passed to the Dataproc API. Let me know. ( I almost dropped IT when I was 14 because of a missing |
|
@ocsig I can't thank you enough for being so patient with me. It worked properly this time. I have PRESTO installation in the dataproc cluster. Do you want me to delete the unneccesary comments above, so that someone else would find it easy to look for the solution. |
|
I was happy to help, no need to delete the comments, the debugging steps are important as well. Maybe put a short TL:DR; on the top of your opening comment. |
|
Thank you once again. :) I added the link to the solution comment at the top. |
[TL;DR] Here is the solution to the below problem: #546 (comment)
I tried creating a Presto dataproc cluster using the
optionalComponentsfield under the software-config, but I observed only PRESTO is not getting installed. Rest all other components are successfully getting installed from the below template.Besides, I see Presto is not in beta as well, so what could be the possible solution here?
The text was updated successfully, but these errors were encountered: