DetachedInstanceError when harvesting CKAN #151

florianm · 2015-08-18T06:08:35Z

When harvesting a CKAN instance (source install v2.5.0a as of July 2015) into another CKAN instance (datacats using latest master, v2.5.0a) with a custom ckanext-scheming dataset schema, I'm running into DetachedInstanceError indicating that the db conection via SQLAlchemy changes sessions.

DetachedInstanceError('Instance <User at 0x7fdf11ae5e50> is not bound to a Session; attribute refresh operation cannot proceed',)

My config:

{
 "api_version": 1,
 "user": "harvest",
 "remote_groups": "create",
 "remote_orgs": "create",
 "override_extras": true,
 "read_only": true,
 "force_all": true
}

I've created a sysadmin user called "harvest", same error as with my own sysadmin account.

I haven't seen a DetachedInstanceError since mentioned in the dev meeting archives a year ago.
Has anyone seen this error lately, or is harvesting only broken on my end?

The text was updated successfully, but these errors were encountered:

raphaelstolt · 2015-09-02T15:23:57Z

Also getting this error constantly, though my CKAN instance version is 2.3, and harvesting seems to be seriously broken.

My config settings are:

 {
    "remote_groups": "only_local", 
    "force_all": true, 
    "user": "harvest", 
    "api_version": 1
}

And a local group catalogue is present.

davidread · 2015-09-02T15:47:03Z

You probably know this, but just in case: when you get an ORM object (e.g. with model.Package.get(id) or model.Session.query(model.Package).first()) then you can use it until you do a model.Session.commit()/close()/remove(). After that you get a DetachedInstanceError if you try and use the ORM object again. That is the most likely cause.

I suggest you put in a breakpoint or use prints to find out where this is happening and fix it.

raphaelstolt · 2015-09-08T09:42:05Z

Seems like adding 'defer_commit': True to the context dictionary in a custom harvester prevents the DetachedInstanceError from being raised. Can someone confirm this?

florianm · 2015-09-08T11:17:01Z

Dave, thanks for the info - didn't know that. As I was using a vanilla ckan
harvester it didn't know I had to intervene at that level. Let me double
check that it wasn't due to some mods on my end.

Raphael, thanks I'll try that out as soon as I can, gonna be early next
week!
On 08/09/2015 7:42 pm, "Raphael Stolt" notifications@github.com wrote:

Seems like adding 'defer_commit': True to the context dictionary prevents
the DetachedInstanceError from being raised. Can someone confirm this?

—
Reply to this email directly or view it on GitHub
#151 (comment)
.

davidread · 2015-09-08T11:32:01Z

Please everyone, let's help each other more - if no-one tries to fix
problems like this (or pay someone to do it), and contributes back as PRs,
nothing will improve. There is no magic central funding for CKAN that
guarantees things work.

Deferring the commit sounds like a bad solution that complicates things. It
will break-down at some point in a confusing way. I'd encourage you to take
the time to find the actual place where the commit is done and the object
is used after it.

On 8 September 2015 at 12:17, Florian Mayer notifications@github.com
wrote:

Dave, thanks for the info - didn't know that. As I was using a vanilla ckan
harvester it didn't know I had to intervene at that level. Let me double
check that it wasn't due to some mods on my end.

Raphael, thanks I'll try that out as soon as I can, gonna be early next
week!
On 08/09/2015 7:42 pm, "Raphael Stolt" notifications@github.com wrote:

Seems like adding 'defer_commit': True to the context dictionary prevents
the DetachedInstanceError from being raised. Can someone confirm this?

—
Reply to this email directly or view it on GitHub
<
https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138497118>
.

—
Reply to this email directly or view it on GitHub
#151 (comment)
.

florianm · 2015-09-08T11:42:06Z

Nope, not deferring, only on holidays this week and banned from coding by
executive order of wife. Will work on it first thing when I'm back - hope
that's enough to avoid the wrath of ckan.
On 08/09/2015 9:32 pm, "David Read" notifications@github.com wrote:

Please everyone, let's help each other more - if no-one tries to fix
problems like this (or pay someone to do it), and contributes back as PRs,
nothing will improve. There is no magic central funding for CKAN that
guarantees things work.

Deferring the commit sounds like a bad solution that complicates things. It
will break-down at some point in a confusing way. I'd encourage you to take
the time to find the actual place where the commit is done and the object
is used after it.

On 8 September 2015 at 12:17, Florian Mayer notifications@github.com
wrote:

Dave, thanks for the info - didn't know that. As I was using a vanilla
ckan
harvester it didn't know I had to intervene at that level. Let me double
check that it wasn't due to some mods on my end.

Raphael, thanks I'll try that out as soon as I can, gonna be early next
week!
On 08/09/2015 7:42 pm, "Raphael Stolt" notifications@github.com wrote:

Seems like adding 'defer_commit': True to the context dictionary
prevents
the DetachedInstanceError from being raised. Can someone confirm this?

—
Reply to this email directly or view it on GitHub
<

https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138497118>

.

—
Reply to this email directly or view it on GitHub
<
https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138521331>
.

—
Reply to this email directly or view it on GitHub
#151 (comment)
.

florianm · 2015-09-08T11:46:32Z

Wait I think I misunderstood what Dave meant with "deferring". Nevertheless
the WiFi up here is too shaky to ssh into my ckan.
On 08/09/2015 9:42 pm, "Florian May" florian.wendelin.mayer@gmail.com
wrote:

Nope, not deferring, only on holidays this week and banned from coding by
executive order of wife. Will work on it first thing when I'm back - hope
that's enough to avoid the wrath of ckan.
On 08/09/2015 9:32 pm, "David Read" notifications@github.com wrote:

Please everyone, let's help each other more - if no-one tries to fix
problems like this (or pay someone to do it), and contributes back as PRs,
nothing will improve. There is no magic central funding for CKAN that
guarantees things work.

Deferring the commit sounds like a bad solution that complicates things.
It
will break-down at some point in a confusing way. I'd encourage you to
take
the time to find the actual place where the commit is done and the object
is used after it.

On 8 September 2015 at 12:17, Florian Mayer notifications@github.com
wrote:

Dave, thanks for the info - didn't know that. As I was using a vanilla
ckan
harvester it didn't know I had to intervene at that level. Let me double
check that it wasn't due to some mods on my end.

Raphael, thanks I'll try that out as soon as I can, gonna be early next
week!
On 08/09/2015 7:42 pm, "Raphael Stolt" notifications@github.com
wrote:

Seems like adding 'defer_commit': True to the context dictionary
prevents
the DetachedInstanceError from being raised. Can someone confirm this?

—
Reply to this email directly or view it on GitHub
<

#151 (comment)

.

—
Reply to this email directly or view it on GitHub
<
#151 (comment)

.

—
Reply to this email directly or view it on GitHub
#151 (comment)
.

davidread · 2015-09-08T12:13:07Z

Sorry Florian, I overreacted. I just want to guard against people thinking
this stuff should work flawlessly. Of course it's good to ask around like
you have. I just want to encourage everyone to get their hands dirty and
fix things with PRs too, and I know you are one of the best examples of
that.

On 8 September 2015 at 12:46, Florian Mayer notifications@github.com
wrote:

Wait I think I misunderstood what Dave meant with "deferring". Nevertheless
the WiFi up here is too shaky to ssh into my ckan.
On 08/09/2015 9:42 pm, "Florian May" florian.wendelin.mayer@gmail.com
wrote:

Nope, not deferring, only on holidays this week and banned from coding by
executive order of wife. Will work on it first thing when I'm back - hope
that's enough to avoid the wrath of ckan.
On 08/09/2015 9:32 pm, "David Read" notifications@github.com wrote:

Please everyone, let's help each other more - if no-one tries to fix
problems like this (or pay someone to do it), and contributes back as
PRs,
nothing will improve. There is no magic central funding for CKAN that
guarantees things work.

Deferring the commit sounds like a bad solution that complicates things.
It
will break-down at some point in a confusing way. I'd encourage you to
take
the time to find the actual place where the commit is done and the
object
is used after it.

On 8 September 2015 at 12:17, Florian Mayer notifications@github.com
wrote:

Dave, thanks for the info - didn't know that. As I was using a vanilla
ckan
harvester it didn't know I had to intervene at that level. Let me
double
check that it wasn't due to some mods on my end.

Raphael, thanks I'll try that out as soon as I can, gonna be early
next
week!
On 08/09/2015 7:42 pm, "Raphael Stolt" notifications@github.com
wrote:

Seems like adding 'defer_commit': True to the context dictionary
prevents
the DetachedInstanceError from being raised. Can someone confirm
this?

—
Reply to this email directly or view it on GitHub
<

#151 (comment)

.

—
Reply to this email directly or view it on GitHub
<

#151 (comment)

.

—
Reply to this email directly or view it on GitHub
<
https://github.com/ckan/ckanext-harvest/issues/151#issuecomment-138526001>
.

—
Reply to this email directly or view it on GitHub
#151 (comment)
.

davidread · 2015-10-28T19:42:08Z

BTW I've just added a testing framework for harvesters, which might help you identify the problem, if it is to do with your specific harvester. And if you can write a failing test it helps other people reproduce and help with the problem. See this example for the CKAN harvester:

https://github.com/ckan/ckanext-harvest/blob/master/ckanext/harvest/tests/harvesters/test_ckanharvester.py#L74-L89

florianm · 2015-10-28T22:44:10Z

@davidread thanks for the testing framework, I'll give it a whirl!

FYI my particular harvesting requirements ended up being highly customised and co-evolving with the harvesting process, so I scripted the process in the first instance, here's the code for reference. I'm breaking away from the ckanext-harvesting paradigm in two incompatible ways:

custom schema on the receiving end - ckanext-scheming forbids free-form "extra" fields, ckanext-harvesting creates them
what will become resources on one dataset are distributed over different endpoints - e.g., a WMS endpoint contains a few hundred layers, a corresponding WFS endpoint contains a few layers matching their WMS counterparts, plus other stand-alone layers. There's multiple WMS/WFS endpoint pairs, and I can't predict duplication between them. I ended up writing my custom "upsert dataset" method that merges new resources into existing datasets or creates new datasets.

davidread · 2015-10-28T23:12:40Z

Interesting point about ckanext-scheming and harvesting - I've not used scheming yet but I guess it means adapting whatever harvester for that.

The WMS/WFS thing looks annoying to deal with.

I quickly looked at your 'harvest' scripts and I guess you could simply drop them into a harvester gather_stage and they'd run fine, upsert and all. Then the harvest framework would give you a web front-end for viewing and running them, could schedule runs etc. but you'd lose having them in a python notebook which is easier to toy with when necessary. (The harvest framework assumes you'd create datasets across a single organization, but if you hid or ignored that field I don't think there'd be a problem.)

DieterDePaepe · 2015-11-09T13:53:46Z

Encountered the same error. My CKAN is version 2.4.0 running on an ubuntu, not running any customisations. Strangely, harvesting worked for several months, until just now. I'll list what I did, in case it might help find the origin problem. I'm not a Python developer, so I can't trace this down myself.

Had several harvesters, all set to manual mode.
Try to add new harvester with same URL as a present one (I wanted to verify whether the content had changed of the DCAT being harvested). Received error that a harvester already existed with that URL.
Change the URL of the existing harvester by appending a random string.
Save the new harvester, using the now available URL.
Try harvesting by running the harvester run script.
Crash occurs.

Below is the error stacktrace. This error occured when I started harvesting, and now occurs whenever I try to start the fetch_consumer script.

(default)dieter@dieter-VB-CKAN:~$ paster --plugin=ckanext-harvest harvester fetch_consumer --config=/etc/ckan/default/production.ini 
2015-11-09 14:32:14,214 DEBUG [ckanext.harvest.model] Harvest tables defined in memory
2015-11-09 14:32:14,216 DEBUG [ckanext.harvest.model] Harvest tables already exist

2015-11-09 14:32:14,329 DEBUG [ckanext.harvest.queue] Fetch queue consumer registered
2015-11-09 14:32:14,330 INFO  [ckanext.harvest.queue] Received harvest object id: 373e3b8b-14c9-4abd-978b-ebd3544b06d1
2015-11-09 14:32:14,360 DEBUG [ckanext.dcat.harvesters.rdf] In DCATRDFHarvester import_stage
2015-11-09 14:32:14,483 DEBUG [ckanext.harvest.harvesters.base] Update validation Error: {'Resources': 'Package resource(s) invalid'}
2015-11-09 14:32:14,488 INFO  [ckanext.harvest.queue] Received harvest object id: 95004674-f3c2-4681-930b-d2365c9cac0d
2015-11-09 14:32:14,501 DEBUG [ckanext.dcat.harvesters.rdf] In DCATRDFHarvester import_stage
Traceback (most recent call last):
  File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
    load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
    invoke(command, command_name, options, args[1:])
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
    exit_code = runner.run(args)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
    result = self.command()
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 144, in command
    fetch_callback(consumer, method, header, body)
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py", line 305, in fetch_callback
    fetch_and_import_stages(harvester, obj)
  File "/usr/lib/ckan/default/src/ckanext-harvest/ckanext/harvest/queue.py", line 322, in fetch_and_import_stages
    success_import = harvester.import_stage(obj)
  File "/usr/lib/ckan/default/src/ckanext-dcat/ckanext/dcat/harvesters/rdf.py", line 273, in import_stage
    existing_dataset = self._get_existing_dataset(harvest_object.guid)
  File "/usr/lib/ckan/default/src/ckanext-dcat/ckanext/dcat/harvesters/rdf.py", line 82, in _get_existing_dataset
    return p.toolkit.get_action('package_show')({}, {'id': datasets[0][0]})
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py", line 429, in wrapped
    result = _action(context, data_dict, **kw)
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/action/get.py", line 927, in package_show
    _check_access('package_show', context, data_dict)
  File "/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py", line 296, in check_access
    data_dict)
  File "/usr/lib/ckan/default/src/ckan/ckan/authz.py", line 153, in is_authorized
    user = _get_user(username)
  File "/usr/lib/ckan/default/src/ckan/ckan/authz.py", line 118, in _get_user
    if c.userobj and c.userobj.name == username:
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 233, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 577, in get
    value = callable_(state, passive)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/state.py", line 422, in __call__
    self.manager.deferred_scalar_loader(self, toload)
  File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/loading.py", line 563, in load_scalar_attributes
    (state_str(state)))
sqlalchemy.orm.exc.DetachedInstanceError: Instance <User at 0x5370450> is not bound to a Session; attribute refresh operation cannot proceed

Note that the invalid resource error is probably related to one of my harvesters using an incorrect DCAT feed.

raphaelstolt · 2015-11-09T14:24:03Z

When going down the rabbit hole to fix or better said avoid this error, I noticed when disabling the core CKAN recline_view plugin or replacing it with recline_grid_view the error went away. Don't really know how these intertwine but worked for me.

DieterDePaepe · 2015-11-09T14:58:24Z

How exactly do you disable/replace that plugin?
For reference, my extentions are, as listed on <host>/api/util/status: "stats", "text_view", "image_view", "recline_view", "harvest", "ckan_harvester", "dcat", "dcat_rdf_harvester", "dcat_json_harvester", "dcat_json_interface"

Further update:
Since I couldn't immediately figure out how to fix that error, I reverted my VM that was running CKAN. It was an almost fresh install. I retried to add the harvesting URL, and immediately encountered the same issue again. Perhaps it has to do with the DCAT that is being read?

I also noticed that the harvester fetch_consumer command keeps crashing as long as the harvest job is running (as seen on <host>/harvest/admin/<job-name>. Pressing Clear on that page allows the harvester fetch_consumer to start again.

raphaelstolt · 2015-11-09T15:01:05Z

Simply by modifiying the plugins list (i.e. ckan.plugins) in /etc/ckan/default/production.ini.

DieterDePaepe · 2015-11-10T15:33:29Z

Final update: I have a situation where I can recreate the error. It is related to the dcat file (my dcat file is missing some necessairy attributes), but I can't narrow it down. I managed to find a single dataset entry in the DCAT that crashed it, only to work it when I reinitialized the VM.

When starting from a completely fresh CKAN installation, I can't reproduce the error, so I guess it has to do with the data that is already present in the DB.

However, I'm not sure how to further investigate this error. My Python-fu is limited, and I have no idea how to debug something that interacts between 3 python processes. If someone gives me some pointers (within reasonable time of this post), I can gather more data.

davidread · 2015-11-10T16:31:09Z

@DieterDePaepe Would you providing the DCAT file? Either with a link or putting it in a pastebin, so I can try creating a test?

The stacktrace shows the problem occurs when the harvester's import stage, when tries to package_show the pre-existing dataset and that tries to access c.userobj. Now c.userobj is initialized in BaseController, but I'm struggling to see how that would have been run by a harvester. I can't get it to initialize running the existing DCAT harvester tests.

BTW did disabling the recline_view plugin make the error go away?

davidread · 2015-11-10T16:35:34Z

@DieterDePaepe You could also try to see what is initializing c.userobj, by inserting this breakpoint into ckan/ckan/lib/base.py around line 281:

             # model directly.
   +         import pdb; pdb.set_trace()
             if c.user and not c.userobj:

and then run the fetch_consumer. Hopefully it should give you a prompt and you can type bt to give you the stack. Could you try that and let me know what the stack shows?

DieterDePaepe · 2015-11-10T20:55:12Z

DCAT feed that causes crash for me can be found at http://pastebin.com/9j9amRxg
Note the missing DownloadURL in each dataset.

As I mentioned before I tried to narrow down the DCAT to produce the same error. I ended up with the 2nd listed dataset (about crime). But since this smaller DCAT didn't cause an error when I tried again, I'm not sure it's strictly related to that one.

Haven't tried yet to disable the recline_view plugin. Will try that and the debug command soon.

DieterDePaepe · 2015-11-12T10:56:04Z

I tried disabling the recline_view plugin and tried replacing it with recline_grid_view. After restarting apache and reloading the harvesters, the error was still present.

Adding the python debugger: after adding it, the ckan website gives an internal error, the logs mention a BdbQuit error, which according to google occurs when the debugger cannot get i/o for interaction. So that makes sense and shouldn't interfere with the debugging of the harvester. However, when running the run command, I get the same DetachedInstanceError as before, no debugger. Perhaps this is because I am running a paster command?

The commands I use to run harvesting are (each in a different console that I keep open):

(default)dieter@dieter-VB-CKAN:/usr/lib/ckan/default/src/ckanext-dcat$ paster --plugin=ckanext-harvest harvester fetch_consumer --config=/etc/ckan/default/production.ini 

(default)dieter@dieter-VB-CKAN:~$ paster --plugin=ckanext-harvest harvester gather_consumer --config=/etc/ckan/default/production.ini 

(default)dieter@dieter-VB-CKAN:~$ paster --plugin=ckanext-harvest harvester run --config=/etc/ckan/default/production.ini

metaodi · 2016-09-15T15:31:24Z

Any update on this issue?

We start to see this errors on one of our servers (it's actually on our test server, fortunately the production works). But they use the same data and have the same configuration.

The Instance <User at 0x3410450> is not bound to a Session error in a custom harvester in the import_stage, in a call to organization_show, to check if an organization exists. It's the first time the context is used. I thought it might be related to 5e1512f, but copying the context didn't help.

amercader · 2016-09-20T09:43:18Z

Does this give any clues? ckan/ckan#1714 (comment)

Sorry, it's a long read but very thorough and it fixed a similar problem I had once with spatial harvesters, so might be worth checking.

metaodi · 2016-10-12T12:30:53Z

@amercader after an extended debugging session, we were finally able to pin down this issue. It was a very unfortunate chain of circumstances that led to this error:

TL;DR:

We use a custom harvester to update datasets, that were not updated since CKAN 2.2
resource_update makes some nasty stuff to the given context and we didn't make new copies for all our calls
The error handling in resource_update and our custom code was less than ideal

Long story for further reference:

Our custom harvester has the requirement to not update the package metadata, but only updated the ressources (because it's possible that the package metadata is changed manually after the harvester runs). Therefore we make calls to resource_create and resource_update, without caring about the package itself.

This failed for most of our packages, i.e. we got in the except block with a KeyError for the key 'resources'. After some looking around, we found the code responsible to this KeyError:

 try:
    context['defer_commit'] = True
    context['use_cache'] = False
    _get_action('package_update')(context, pkg_dict)
    context.pop('defer_commit')
except ValidationError, e:
    errors = e.error_dict['resources'][-1]
    raise ValidationError(errors)

Here comes the interesting part for the DetachedInstanceError. Because we were guilty of not create a new context for each of our calls, this code changed our context. i.e. _get_action() would raise an exception, leave context['defer_commit'] = True in the context for all subsequent calls.

To add more confusion, the raised exception does not contain a resources key on the error_dict attribute, therefore raising a KeyError. I don't know anything about that. But in the end the actual ValidationError was that we used a key on the package in as extra fields (schema error). This was because those datasets were not changed since CKAN 2.2 and therefore were still saved in the old format.

Solution

We ended up adding a little script to "migrate" the old packages to get rid of the extras. This leads to the situation, that we don't get a ValidationError anymore, and therefore everything runs smoothly.

The problem, that the code in resource_update and resource_create could cause a corrupt context and/or throw KeyErrors in an except block is still not solved. I don't know exactly, how this code should behave. But most probaly the context.pop('defer_commit') should be in a finally block, so that the context is restored, even if there was a ValidationError. I'm open to create PR for this, but this is probably not the repository to have this discussion anyway.

I just wanted to leave this here, in case someone needs some more hints to debug this.

amercader · 2016-10-12T14:17:09Z

@metaodi Really glad to hear that you were successful after what it sounds like an intense debugging session :)

Legacy harvesters might have partial blame, but the KeyError on resource_update update is terrible and several people have been bitten by it. There is a PR to provide a better handling, it's just lacking tests. I'll try to find time to write one or feel free to jump on it if you have some: ckan/ckan#3027

MaciejJanowski · 2018-03-21T11:56:30Z

I just installed the new release of CKAN (2.7.3) and encountered same error

INFO  [ckanext.harvest.queue] Received harvest object id: fd026470-dd37-408f-9b57-e819c52f70bf
DEBUG [ckanext.harvest.harvesters.ckanharvester] In CKANHarvester import_stage
DEBUG [ckanext.harvest.model] Instance <User at 0x7f70731823d0> is not bound to a Session; attribute refresh operation cannot proceed

I followed the whole process of installing ckan, datastore, datapusher and harvester.
I am trying to harvest: https://data.smartdublin.ie/.
I tested different sources and the same error occurs. Please advise me what I did wrong or what shall I do to make everything work properly.

florianm · 2018-03-22T02:07:58Z

When I had to migrate data into CKAN, I gave up on the harvester, and used the CKAN API from an iPython notebook: https://github.com/datawagovau/harvesters (same link as in my comment above)

Harvesters are great when running as a scheduled task, but introduce one extra level of complexity which can conflict with other customisations (e.g. ckanext-scheming overriding "extra" fields). Simply whacking the API with a wrench until the data goes in always works.
While this might be a workaround in some cases, I support Dave's advice of actually trying to find a fix.

davidread · 2018-03-23T12:49:44Z

@MaciejJanowski you can discount datapusher and datastore. The problem you're seeing is with ckanharvester.

A bit about your context might help us think about this:

operating system
method of installing CKAN (docker, package or source)
queue running on Redis or RabbitMQ?
are you running gather_consumer and fetch_consumer just in terminals or with supervisor or something?
I assume you triggering the harvest with paster harvester run. Have you tried testing it with paster harvester run_test? If so, does that work?

MaciejJanowski · 2018-03-23T19:24:53Z

@davidread

Operating system: Ubuntu 16.04 64-bit
I installed CKAN from source (CKAN v2.7.3)
Queue running on redis
I tried running it via supervisor and through terminals
On earlier instances I installed just run fetch and gather consumer. I did the same this time.

davidread · 2018-04-01T20:53:14Z

@MaciejJanowski thanks for this. All sounds standard enough.

The tests pass, so I I guess the next step is for you or another developer to investigate why it happens and/or reproduce it in a test. You could use print statements or pdb to find out when the User object is instantiated, when its tried to be used and it's found to be detached, and then track down when it gets detached. At least you're seeing it happen every time, which helps(!)

pduchesne · 2019-04-26T10:49:52Z

I'm hitting the same problem while using the CSW harvester, using

ckan at e838541d (2018-12-12)
ckanext-harvest at 65e3163 (2018-03-03)
a branch of ckanext-spatial at f805d412 (2017-09-14)
It also uses scheming.

FWIW, here's what I observe and how I fixed it, although I'm far from understanding what's exactly happening.

The root cause is a harvested package that fails to validate, causing an error and a rollback of the session (see https://github.com/ckan/ckan/blob/e838541d/ckan/logic/action/create.py#L175).
This apparently causes the userobj in ckan.common.c to become detached.
As a result, when processing the next harvested object, a call to is_authorized (in my case to get the license list : https://github.com/ckan/ckanext-spatial/blob/f805d412/ckanext/spatial/harvesters/base.py#L294) chokes on the userobj being detached (https://github.com/ckan/ckan/blob/e838541d/ckan/authz.py#L119).

I fixed it by checking if the userobj is still attached :

if c.userobj and c.userobj.Session.object_session(c.userobj) and c.userobj.name == username:
            return c.userobj

That's not a satisfying fix; i'd rather understand why this userobj gets detached in the first place or whether it should be re-attached, but that's as far as I can go.

florianm mentioned this issue Aug 18, 2015

Implement catalogue federation datawagovau/pandora#9

Closed

florianm mentioned this issue Aug 29, 2015

Integrate ckan_harvester with ckanext-scheming #153

Open

This was referenced Feb 5, 2018

Remove returning stored site user #311

Closed

Validate dataset before create or update #312

Open

metaodi mentioned this issue Mar 21, 2018

[ckanext.harvest.model] Instance is not bound to a Session #317

Closed

bonnland mentioned this issue May 19, 2021

Avoid Sqlalchemy errors from prior harvest rollbacks #445

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DetachedInstanceError when harvesting CKAN #151

DetachedInstanceError when harvesting CKAN #151

florianm commented Aug 18, 2015

raphaelstolt commented Sep 2, 2015

davidread commented Sep 2, 2015

raphaelstolt commented Sep 8, 2015

florianm commented Sep 8, 2015

davidread commented Sep 8, 2015

florianm commented Sep 8, 2015

florianm commented Sep 8, 2015

davidread commented Sep 8, 2015

davidread commented Oct 28, 2015

florianm commented Oct 28, 2015

davidread commented Oct 28, 2015

DieterDePaepe commented Nov 9, 2015

raphaelstolt commented Nov 9, 2015

DieterDePaepe commented Nov 9, 2015

raphaelstolt commented Nov 9, 2015

DieterDePaepe commented Nov 10, 2015

davidread commented Nov 10, 2015

davidread commented Nov 10, 2015

DieterDePaepe commented Nov 10, 2015

DieterDePaepe commented Nov 12, 2015

metaodi commented Sep 15, 2016

amercader commented Sep 20, 2016

metaodi commented Oct 12, 2016

amercader commented Oct 12, 2016

MaciejJanowski commented Mar 21, 2018

florianm commented Mar 22, 2018 •

edited

davidread commented Mar 23, 2018

MaciejJanowski commented Mar 23, 2018

davidread commented Apr 1, 2018

pduchesne commented Apr 26, 2019

DetachedInstanceError when harvesting CKAN #151

DetachedInstanceError when harvesting CKAN #151

Comments

florianm commented Aug 18, 2015

raphaelstolt commented Sep 2, 2015

davidread commented Sep 2, 2015

raphaelstolt commented Sep 8, 2015

florianm commented Sep 8, 2015

davidread commented Sep 8, 2015

florianm commented Sep 8, 2015

florianm commented Sep 8, 2015

davidread commented Sep 8, 2015

davidread commented Oct 28, 2015

florianm commented Oct 28, 2015

davidread commented Oct 28, 2015

DieterDePaepe commented Nov 9, 2015

raphaelstolt commented Nov 9, 2015

DieterDePaepe commented Nov 9, 2015

raphaelstolt commented Nov 9, 2015

DieterDePaepe commented Nov 10, 2015

davidread commented Nov 10, 2015

davidread commented Nov 10, 2015

DieterDePaepe commented Nov 10, 2015

DieterDePaepe commented Nov 12, 2015

metaodi commented Sep 15, 2016

amercader commented Sep 20, 2016

metaodi commented Oct 12, 2016

TL;DR:

Long story for further reference:

Solution

amercader commented Oct 12, 2016

MaciejJanowski commented Mar 21, 2018

florianm commented Mar 22, 2018 • edited

davidread commented Mar 23, 2018

MaciejJanowski commented Mar 23, 2018

davidread commented Apr 1, 2018

pduchesne commented Apr 26, 2019

florianm commented Mar 22, 2018 •

edited