unable to backup; error message isn't helping..

[manos]

I have backups configured for S3. When I try to create an index, I get this in the log:
Tue Jul 09 23:47:17 GMT 2013 ERROR Building Index (Access Denied)

I'm also using s3 to store the configs, and that's working perfectly. I've created an appropriate IAM ACL for the backup location too.

Locally, the Log Index Dir, is owned by the exhibitor user and is writable (but empty).

I even tried strace -f'ing the JVM to see what was denied. I can't see any open() calls to the log index dir failing.

How can I get more verbose logging? Or, if anyone has ideas about what to check RE: 'Access Denied', I'm all ears :)

TIA

(running 1.4.8)

[manos]

Weird. This morning, I also see this in the logs:
Wed Jul 10 00:36:52 GMT 2013 ERROR Unhandled exception in repeating activity () - re-queueing (Access Denied)
Wed Jul 10 00:36:15 GMT 2013 INFO Cleanup task completed

[Randgalt]

Wed Jul 10 00:36:52 GMT 2013 ERROR Unhandled exception in repeating activity () - re-queueing (Access Denied)

Those come from the backup task usually. It's trying to backup a log file and ZK deletes it (or possibly the cleanup task deletes it). I have a note to not report these harmless errors.

-JZ

On Jul 10, 2013, at 9:07 AM, Charlie Schluting notifications@github.com wrote:

Weird. This morning, I also see this in the logs:
Wed Jul 10 00:36:52 GMT 2013 ERROR Unhandled exception in repeating activity () - re-queueing (Access Denied)
Wed Jul 10 00:36:15 GMT 2013 INFO Cleanup task completed

—
Reply to this email directly or view it on GitHub.

[Randgalt]

Tue Jul 09 23:47:17 GMT 2013 ERROR Building Index (Access Denied)

I haven't seen that before. What that implies, though, is that the "Log Index Dir" in your Exhibitor config is inaccessible. What version of *nix is this? Have you checked the perms on that directory?

-JZ

[Randgalt]

In any event, I'll try to add a better logging messaging for this.

On Jul 9, 2013, at 5:15 PM, Charlie Schluting notifications@github.com wrote:

How can I get more verbose logging? Or, if anyone has ideas about what to check RE: 'Access Denied', I'm all ears :)

[manos]

Thanks - that's what I expected.
This is on ubuntu 10.04 (lucid). Exhibitor and zookeeper run as the user 'zookeeper', uid 999.

Log Index Dir is set to /data/exhibitor/, and that directory is owned by zookeeper. I even tried mode 0777 to test if something weird was going on, like some process was dropping privs. Nada.

[Randgalt]

OK - I'll be building a new Exhibitor in the next day or so and I'll add more logging. Hopefully, that will help get to the bottom of this.

-JZ

On Jul 10, 2013, at 9:29 AM, Charlie Schluting notifications@github.com wrote:

Thanks - that's what I expected.
This is on ubuntu 10.04 (lucid). Exhibitor and zookeeper run as the user 'zookeeper', uid 999.

Log Index Dir is set to /data/exhibitor/, and that directory is owned by zookeeper. I even tried mode 0777 to test if something weird was going on, like some process was dropping privs. Nada.

—
Reply to this email directly or view it on GitHub.

[Randgalt]

The new version of Exhibitor is available. Let me know if the additional logging has helped narrow the issue.

[manos]

Thanks!

Turns out, I now see 'S3 access denied' errors on stdout (of the exhibitor process). But FYI, the Web interface still just says "Access denied" without identifying what it's referring to.

Which makes no sense... I'm sure the IAM role is correct. And config syncing to a different bucket (similar policy) is working perfectly. I'm going to ask AWS - BTW love that the transaction ID is in the error message :)

[Randgalt]

At this point I believe this to be an AWS config issue of some kind and will close the issue. Please re-open if you find something else.

[manos]

Hi @Randgalt - turns out, Exhibitor tries to list the bucket before even starting indexing.

So, if someone (like me!) configures a prefix key, and then uses IAM to grant access only to that path, it will fail. Exhibitor apparently needs access to list the bucket contents too. (i.e. the IAM policy must grant access to the bucket name - with NO trailing slash! - as well as the full path that's being used for backups).

I would suggest that Exhibitor only needs to list the bucket+key_prefix, to check for existing backups..

Thanks for your help!

[mrwilby]

Same issue. 403 because exhibitor attempts to access outside the key-prefix.

Once I edited my IAM policy to remove the key prefix restriction, exhibitor worked.

This is IMO a legitimate bug...

[Randgalt]

A PR with the fix would be appreciated.