Permission denied when trying to download NCDC data from S3

[brunitto]

Hi.

I am trying do download the NCDC data from S3, as defined in:

http://hadoopbook.com/code.html

Got the following error:

17/04/17 18:34:57 INFO tools.DistCp: Input Options: DistCpOptions{atomicCommit=false, syncFolder=false, deleteMissing=false, ignoreFailures=false, maxMaps=20, sslConfigurationFile='null', copyStrategy='uniformsize', sourceFileListing=null, sourcePaths=[s3n://hadoopbook/ncdc/all], targetPath=input/ncdc/all, targetPathExists=false, preserveRawXattrs=false}
17/04/17 18:34:57 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
17/04/17 18:34:57 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
17/04/17 18:35:00 ERROR tools.DistCp: Exception encountered
org.apache.hadoop.security.AccessControlException: Permission denied: s3n://hadoopbook/ncdc/all_$folder$
        at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:449)
        at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.processException(Jets3tNativeFileSystemStore.java:427)
        at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.handleException(Jets3tNativeFileSystemStore.java:411)
        at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:181)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:601)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
        at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
        at org.apache.hadoop.fs.s3native.$Proxy10.retrieveMetadata(Unknown Source)
        at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:483)
        at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
        at org.apache.hadoop.fs.Globber.glob(Globber.java:252)
        at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1657)
        at org.apache.hadoop.tools.GlobbedCopyListing.doBuildListing(GlobbedCopyListing.java:77)
        at org.apache.hadoop.tools.CopyListing.buildListing(CopyListing.java:84)
        at org.apache.hadoop.tools.DistCp.createInputFileListing(DistCp.java:382)
        at org.apache.hadoop.tools.DistCp.createAndSubmitJob(DistCp.java:181)
        at org.apache.hadoop.tools.DistCp.execute(DistCp.java:153)
        at org.apache.hadoop.tools.DistCp.run(DistCp.java:126)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.tools.DistCp.main(DistCp.java:430)
Caused by: org.jets3t.service.impl.rest.HttpException
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:519)
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRequest(RestStorageService.java:281)
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.performRestHead(RestStorageService.java:942)
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectImpl(RestStorageService.java:2148)
        at org.jets3t.service.impl.rest.httpclient.RestStorageService.getObjectDetailsImpl(RestStorageService.java:2075)
        at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:1093)
        at org.jets3t.service.StorageService.getObjectDetails(StorageService.java:548)
        at org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(Jets3tNativeFileSystemStore.java:174)
        ... 19 more
hadoop@single-node:~$

[brunitto]

Tried within an EC2 instance in us-east-1 region.

[aliyevnb]

tried it with emr cluster in us-east-1, looks like permission issue. aws cli command return following (snippet)
..
download failed: s3://hadoopbook/ncdc/all/2001.gz to ../../mnt/2001.gz An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
..

[aliyevnb]

It's a wilde guess but there is a most likely access is restricted to a AWS us-east-1 subnets on S3 GetObjects, because I can list objects with no issues. I assume bucket policy has restrictions on us-east-1 ip range. Latest us-east-1 public ip range available here:

curl https://ip-ranges.amazonaws.com/ip-ranges.json | jq '.prefixes[] | select(.service | contains("EC2")) | select(.region | contains("us-east-1"))'

[dahejin]

Remove me from this list. From: Bruno Moyle <notifications@github.com> Sent: Saturday, July 11, 2020 12:50 PM To: tomwhite/hadoop-book <hadoop-book@noreply.github.com> Cc: Subscribed <subscribed@noreply.github.com> Subject: Re: [tomwhite/hadoop-book] Permission denied when trying to download NCDC data from S3 (#30) Closed #30 <#30> . — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#30 (comment)> , or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFRQLMKOGKT2K2VMVZFL66DR3C65XANCNFSM4DH6NENA> . <https://github.com/notifications/beacon/AFRQLMMBKJ3ONZTAMGS6K43R3C65XA5CNFSM4DH6NENKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGO2LFXQXQ.gif>