[NC | NSFS | Glacier] Filter out failed recalls

[tangledbytes]

Explain the changes

This PR fixes a major issue where NooBaa records the failures for retries but does NOT filters out the failed recalls while assigning extended attributes. This will make it seem to NSFS process that the object is restored and it will allow read which will result in a blocking read!

Testing Instructions:

1. Run ./node_modules/.bin/mocha src/test/unit_tests/test_nsfs_glacier_backend.js

• Doc added/updated
• Tests added

[y-sasn]

Hi, Very quick review/comment.

The command "task show -r" displays success or failure counts at the head of each task.
We can't use the "--json" with "-r" unfortunately, so yes, parsing the result seems like a natural choice.
It should only show success/fail for the first column, so parsing the status with success/fail status with line.startsWith("Fail/Start"); seems ok too.

(It might be more concrete to run "eeadm file state" for each of the files given after the task has finished since the information we need is the actual status of the file, not the result of the recall command. Though I guess this could be handled in the future.)

What we need to care about is the definition of the "Success" and "Failed" tasks.
"Success" stands for recalled/already_premigrated/skipped_resident, so that seems ok as is.
"Fail" stands for duplicate/not_found/other_failure. "duplicate" might be an issue depending on how dup files are handled.

If NooBaa removes any dup file requests before handing it to EE's recall, we are good.
If NooBaa does hand dup files to EE, I recommend we add a logic to delete the dups prior to giving it to EE.
If that is not possible, we might need more work on post-proc of the "Fail"ed files.

[tangledbytes]

Hi, Very quick review/comment.

The command "task show -r" displays success or failure counts at the head of each task.

We can't use the "--json" with "-r" unfortunately, so yes, parsing the result seems like a natural choice.

It should only show success/fail for the first column, so parsing the status with success/fail status with line.startsWith("Fail/Start"); seems ok too.

(It might be more concrete to run "eeadm file state" for each of the files given after the task has finished since the information we need is the actual status of the file, not the result of the recall command. Though I guess this could be handled in the future.)

What we need to care about is the definition of the "Success" and "Failed" tasks.

"Success" stands for recalled/already_premigrated/skipped_resident, so that seems ok as is.

"Fail" stands for duplicate/not_found/other_failure. "duplicate" might be an issue depending on how dup files are handled.

If NooBaa removes any dup file requests before handing it to EE's recall, we are good.

If NooBaa does hand dup files to EE, I recommend we add a logic to delete the dups prior to giving it to EE.

If that is not possible, we might need more work on post-proc of the "Fail"ed files.

@y-sasn, if we find a failure which has the error code of "duplicate task" then we don't consider it to be a failure and don't requeue the task. Does that sound okay?

[y-sasn]

@tangledbytes Yes, if one of the other "dup" tasks fails due to a different reason, I'm guessing that that file will be queued. If that's the case I'm guessing we're good.

[tangledbytes]

@tangledbytes Yes, if one of the other "dup" tasks fails due to a different reason, I'm guessing that that file will be queued. If that's the case I'm guessing we're good.

@y-sasn, yes if we find any other failure code then we requeue the item.