Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Don't try to get the file name of purged files #4066

Merged
merged 1 commit into from May 15, 2017

Conversation

Projects
None yet
5 participants
@dpryan79
Copy link
Contributor

commented May 15, 2017

This has been bugging me for a while, but I only just now had a chance to track down the causes.

If one allows users to purge their data then they can often run into problems if they then try to (1) display deleted (including purged in this case) items in a history or (2) click on the information button on a deleted and purged item. The errors are displayed via javascript on the user end, an example of which is below:

"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:45.0) Gecko/20100101 Firefox/45.0",
"onLine": true,
"version": "17.01",
  "xhr": {
    "readyState": 4,
    "responseText": "{\"err_msg\": \"objectstore, _call_method failed: get_filename on <galaxy.model.Dataset object at 0x7f7ecfe7b710>, kwargs: {}\", \"err_code\": 404001}",
    "responseJSON": {
      "err_msg": "objectstore, _call_method failed: get_filename on <galaxy.model.Dataset object at 0x7f7ecfe7b710>, kwargs: {}",
      "err_code": 404001
    },
    "status": 404,
    "statusText": "Not Found"
  },
  "options": {
    "parse": true,
    "silent": true,
    "limit": 500,
    "offset": 0,
    "details": "bbedde4931a34160,bf1fc87dfb22ab7b,c060d876360b529c,239de9509e5f2ad5,031ac30ff5ede248,8a614701a0f1b6a2,3a4220c205f26e2c,17abf85b2306c490,28eb2b64ae0059de,078ee83f6a81d187,487ef6b7ce257489,a0f3c3eaddadcc06,87833dc316faf4b6,910f5492ec37eb15,d895a91075d8a8ce,20f5fb4fc6089119,654b16c6f6aa34ce,d41ff3ffbb639bb5,8682b451162603de,fa448058b8ac2a47,302ddcb06e18bc43,72c46c8eb9871f99,a78e2601656e62e9,915b8d6cc740513a",
    "traditional": true,
    "data": {
      "limit": 500,
      "offset": 0,
      "details":    "bbedde4931a34160,bf1fc87dfb22ab7b,c060d876360b529c,239de9509e5f2ad5,031ac30ff5ede248,8a614701a0f1b6a2,3a4220c205f26e2c,17abf85b2306c490,28eb2b64ae0059de,078ee83f6a81d187,487ef6b7ce257489,a0f3c3eaddadcc06,87833dc316faf4b6,910f5492ec37eb15,d895a91075d8a8ce,20f5fb4fc6089119,654b16c6f6aa34ce,d41ff3ffbb639bb5,8682b451162603de,fa448058b8ac2a47,302ddcb06e18bc43,72c46c8eb9871f99,a78e2601656e62e9,915b8d6cc740513a",
      "order": "hid",
      "v": "dev",
      "q": [
        "visible"
      ],
      "qv": [
        "True"
      ]
    },
    "emulateHTTP": false,
    "emulateJSON": false,
    "textStatus": "error",
    "errorThrown": "Not Found"
  },
  "url": "http://galaxy.ie-freiburg.mpg.de/api/histories/63edcbc9511f9a4c/contents?limit=500&offset=0&details=bbedde4931a34160%2Cbf1fc87dfb22ab7b%2Cc060d876360b529c%2C239de9509e5f2ad5%2C031ac30ff5ede248%2C8a614701a0f1b6a2%2C3a4220c205f26e2c%2C17abf85b2306c490%2C28eb2b64ae0059de%2C078ee83f6a81d187%2C487ef6b7ce257489%2Ca0f3c3eaddadcc06%2C87833dc316faf4b6%2C910f5492ec37eb15%2Cd895a91075d8a8ce%2C20f5fb4fc6089119%2C654b16c6f6aa3    4ce%2Cd41ff3ffbb639bb5%2C8682b451162603de%2Cfa448058b8ac2a47%2C302ddcb06e18bc43%2C72c46c8eb9871f99%2Ca78e2601656e62e9%2C915b8d6cc740513a&order=hid&v=dev&q=visible&    qv=True",
  "model": [],
  "user": {
    "id": "fc24ff678cc48b4e",
    "username": "aline",
    "total_disk_usage": 121603913933,
    "nice_total_disk_usage": "113.3 GB",
    "quota_percent": 56,
    "is_admin": false,
    "preferences": {
      "selected_tool_tags": ""
    },
    "tags_used": [],
    "deleted": false,
    "purged": false,
    "quota": "200.0 GB"
  }
}

In my local instance, I was able to find a few places where this can occur. The simplest is to modify show_params.mako to check for purged files and not attempt to display their paths. I've additionally set some of the Dataset class methods to not try to get file names for purged datasets. At least for my local instance, this allows people to view deleted items (and their parameters) in histories containing purged datasets.

@dannon

This comment has been minimized.

Copy link
Member

commented May 15, 2017

@dpryan79 Whoa, this has been on my list of annoying things to track down for a while, thanks! I'll check this out.

@galaxybot galaxybot added the triage label May 15, 2017

@galaxybot galaxybot added this to the 17.09 milestone May 15, 2017

@nsoranzo

This comment has been minimized.

Copy link
Member

commented May 15, 2017

@dpryan79 Any traceback or message in the logs when that happens?

@dannon

This comment has been minimized.

Copy link
Member

commented May 15, 2017

Related #1765. @dpryan79 Have you talked to @bgruening about this one?

@bgruening

This comment has been minimized.

Copy link
Member

commented May 15, 2017

@dannon I don't think this is the same bug. #1765 is using a wrong function name that does not exists.

@nsoranzo

This comment has been minimized.

Copy link
Member

commented May 15, 2017

@galaxybot test this

@nsoranzo

This comment has been minimized.

Copy link
Member

commented May 15, 2017

@bgruening I think @dannon is right. The tracebacks in #1765 are due to calling NestedObjectStore.get_filename(), which in turn calls NestedObjectStore._call_method(). Here the dataset is searched in the various stores and if not found an ObjectNotFound exception is generated. So calling NestedObjectStore.get_filename() on a purged dataset will very probably cause the exception reported here and in #1765.

@dannon

This comment has been minimized.

Copy link
Member

commented May 15, 2017

This looks like a good manager-level solution. We may want to follow up and see if this can be handled better at the objectstore level, too (if only just making the error(s) more clear).

@dannon dannon merged commit 3da8eb0 into galaxyproject:dev May 15, 2017

5 checks passed

api test Build finished. 276 tests run, 0 skipped, 0 failed.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
framework test Build finished. 148 tests run, 0 skipped, 0 failed.
Details
integration test Build finished. 34 tests run, 0 skipped, 0 failed.
Details
toolshed test Build finished. 579 tests run, 0 skipped, 0 failed.
Details
@dpryan79

This comment has been minimized.

Copy link
Contributor Author

commented May 15, 2017

Sorry, I was away at dinner (you'd be amazed how much sushi a 4 year old can eat).

@nsoranzo: There are oddly no messages in the logs, even at the debug log level. That's one of the more annoying things, since otherwise I'd at least see it when it happens rather than them needing to report it to me.

@dannon: Regarding #1765, I see that all the time and it's not directly related to this since 99% of the time the datasets are not in fact deleted or purged, the NFS is just slow and they haven't been flushed to disk by the cluster nodes yet. But yeah, trying to get a purged object's file name will cause the same sort of problem. I wonder, though, if it's really correct to fix this in the object store. I would think that's just hiding the fact that upstream functions are asking for file names when they shouldn't.

@dpryan79

This comment has been minimized.

Copy link
Contributor Author

commented May 15, 2017

Anyway, thanks for merging!

@dpryan79 dpryan79 deleted the dpryan79:handle_purged_files branch May 15, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.