Upgrade to Plone 4.3.10 #215
Comments
|
I am setting priority to high for this issue. I'd like to see the upgrade and any associated migration etc. tested on staging first (after copying up-to-date copies of filestorage and blobstorage) from production. Then we'll have some idea of what we're up against in terms of a production upgrade. |
|
@paregorios OK, @witekdev will work on this. |
|
Production data has been copied to the staging site (blobs, filestorage and vaytrou data) It seems to me like the python that is on staging is not compiled with the _imaging module that the latest version of PIL needs. @alecpm @cguardia @davisagli can you help out here? |
|
@witekdev try deleting the Pillow egg and rerun buildout. It looks like it was compiled for UCS-4 but the Python does not support it (Python can be compiled with or without support for this way of representing unicode) |
|
@davisagli that worked, the site is up and running. Thoughts? |
|
There is a geographic index which runs in a separate process and is communicated with via TCP. It wasn't running; I ran |
|
thanks @davisagli, I stopped that to copy the vaytrou data over and forgot to restart it. I was clearly too zombified last night to remember. |
|
I ran the upgrade steps on staging, no issues, currently running a catalog update. |
|
ok, maybe @alecpm @cguardia can advise here? When ssh tunneling in directly after a longer period of time I get: When that happens I also see that one of the plone instances has been restarted. Is there a better strategy to do a catalog update on pleaides staging? |
|
Perhaps I should recatalog each index separately? |
|
Something is not right with the re-indexing process. Even the individual re-indexing of the |
|
It's normal for a catalog reindex (whole catalog or individual indexes) to take forever and it will almost always timeout the front end webserver/cache. The reindex needs to wake up every single object in the site and fetch data from them. As a rule, you should run those operations on the admin instance (instance1 on port 8080, you may have to ssh tunnel to access it or add your IP to the firewall list in the ansible config), which won't timeout. You can tail the instance1 log to see how the reindex is progressing, and whether it succeeds or eventually errors out. It's possible (especially on staging) that the instance could run out of RAM during the reindex and get killed by supervisor's memmon or the OS. For staging migrations and reindexes that require a lot of RAM, it sometimes helps to use supervisor to stop all the other instances and the memmon in order to free up RAM and prevent the auto-kill from happening. Keep in mind though a front end timeout will not prevent the operation from continuing, so if it times out you should simply wait and check back later to see if it completed. Don't try to run another reindex after the timeout, because then you will have two instances performing the same long running operation and will inevitably end up with either a out of memory kill or conflict error followed by conflict retries repeating the whole super-long transaction. |
|
Why is the reindex necessary in this case? Is it part of the upgrade or something you're doing "just in case"? David Glick
|
|
@davisagli @alecpm as part of the plone upgrade process, there was a reindexing of the entire catalog. I don't know why, I assumed it was part of the upgrade. The entire dry run was taking forever and timing out so I took the approach of just applying the upgrade steps separately in portal_setup, and then doing the reindexing. |
|
@davisagli @alecpm I'm not sure if that is just standard upgrade boiler plate or if in fact a recatalog was really needed? |
|
When comparing staging indexes against production indexes: There is only one difference: |
|
After recopying the vatrou data from production to staging, all the indexes in the catalog on staging are identical to those on production. |
|
@paregorios The staging site is now ready for you to test. Thanks! |
|
@paregorios although, I'm actually not certain of that, maybe @alecpm can double check. I just blindly assumed that the changes in #198 were to site data, but if they were to code only, they should be there on staging. |
|
There are migration steps that would need to be run to address the issue. Looks like everything is updated. |
|
@alecpm just to verify: you are saying that not only is the correct code on staging and the correct data on staging, but all the needed migration steps have been run as well? |
|
I believe so @skleinfeldt |
|
So it sounds to me like staging has been running with the upgrade for a long time. I guess it's time to start planning to roll this out on production. @skleinfeldt do we need a story for this, or can we just use this ticket? |
|
How about using this ticket @paregorios ? Although open to your suggestion. |
|
fine by me, but let's hold off doing anything with this until the dust settles from the https crossover |
|
dust is now settled from the https crossover; I assume this ticket now/still is ready to be addressed for production. Is that correct? |
|
Correct |
|
@paregorios Production has been patched finally with a lot of help from @davisagli tonight. Happy its done though :) |
|
thanks all! |
Besides the hotfix deployed yesterday #212. In order for Pleiades to take advantage of the latest minor security fixes: https://plone.org/news/2016/minor-plone-security-fixes. It is best for us to upgrade to the latest Plone version ASAP.
The text was updated successfully, but these errors were encountered: