Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Undefined column: 7 ERROR: column c.calendartype does not exist" after upgrade from 16.0.4 to 17.0.0 #17697

Closed
gaya4 opened this issue Oct 26, 2019 · 16 comments
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap bug feature: install and update needs info

Comments

@gaya4
Copy link

gaya4 commented Oct 26, 2019

Steps to reproduce

  1. Upgrade nextcloud via repository
  2. Upgrade nextcloud via webapp-config
  3. Update via browser with button "start update"

Expected behaviour

The update should run successfully.

Actual behaviour

Upgrade fails with error message (in browser):
Repair step: Clean up orphan event and contact data

Doctrine\DBAL\Exception\InvalidFieldNameException: An exception occurred while executing 'SELECT "c"."id" FROM "oc_calendarobjects" "c" LEFT JOIN "oc_calendars" "p" ON "c"."calendarid" = "p"."id" WHERE ("p"."id" IS NULL) AND ("c"."calendartype" = ?)' with params [0]: SQLSTATE[42703]: Undefined column: 7 ERROR: column c.calendartype does not exist LINE 1: ...endarid" = "p"."id" WHERE ("p"."id" IS NULL) AND ("c"."calen... ^

Server configuration

Operating system:
Gentoo
Web server:
Server version: Apache/2.4.41 (Unix)
Server built: Aug 17 2019 06:15:36
Database:
psql (PostgreSQL) 11.5
PHP version:
PHP 7.3.11 (cli) (built: Oct 26 2019 20:02:13) ( ZTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.11, Copyright (c) 1998-2018 Zend Technologies
with Zend OPcache v7.3.11, Copyright (c) 1999-2018, by Zend Technologies
Nextcloud version: (see Nextcloud admin page)
17.0.0
Updated from an older Nextcloud/ownCloud or fresh install:
updated from 16.0.4
Where did you install Nextcloud from:
repository
Signing status:

Signing status
Login as admin user into your Nextcloud and access 
http://example.com/index.php/settings/integrity/failed 
paste the results here.

-> "Nextcloud will be updated to version 17.0.0" -> updater will try to run again.
No login possible.

List of activated apps:

App list contact, calendar, see below: ``` If you have access to your command line run e.g.: sudo -u www-data php occ app:list from within your Nextcloud installation folder ``` sudo -u apache php occ app:list :( The process control (PCNTL) extensions are required in case you want to interrupt long running commands - see http://php.net/manual/en/book.pcntl.php The current PHP memory limit is below the recommended value of 512MB. Nextcloud or one of the apps require upgrade - only a limited number of commands are available You may use your browser or the occ upgrade command to do the upgrade Enabled: - accessibility: 1.3.0 - activity: 2.10.1 - admin_audit: 1.6.0 - calendar: 1.7.1 - cloud_federation_api: 1.0.0 - contacts: 3.1.4 - dav: 1.9.2 - federatedfilesharing: 1.7.0 - files: 1.12.0 - files_frommail: 0.3.0 - files_pdfviewer: 1.6.0 - files_rightclick: 0.14.2 - files_sharing: 1.8.0 - files_trashbin: 1.6.0 - files_videoplayer: 1.6.0 - gallery: 18.4.0 - lookup_server_connector: 1.5.0 - notifications: 2.4.1 - oauth2: 1.5.0 - provisioning_api: 1.7.0 - theming: 1.7.0 - twofactor_backupcodes: 1.6.0 - updatenotification: 1.7.0 - viewer: 1.1.0 - workflowengine: 1.6.0 Disabled: - comments - encryption - federation - files_external - files_versions - firstrunwizard - logreader - nextcloud_announcements - password_policy - privacy - recommendations - serverinfo - sharebymail - support - survey_client - systemtags - text - user_ldap

Nextcloud configuration:

Config report
If you have access to your command line run e.g.:
sudo -u www-data php occ config:list system
from within your Nextcloud installation folder
sudo -u apache php occ config:list system
The process control (PCNTL) extensions are required in case you want to interrupt long running commands - see http://php.net/manual/en/book.pcntl.php
The current PHP memory limit is below the recommended value of 512MB.
Nextcloud or one of the apps require upgrade - only a limited number of commands are available
You may use your browser or the occ upgrade command to do the upgrade
{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "uri-international.de",
            "share.uri-international.de",
            "192.168.202.254"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "pgsql",
        "version": "16.0.4.1",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbtableprefix": "oc_",
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "logtimezone": "UTC",
        "installed": true,
        "theme": "",
        "loglevel": 2,
        "maintenance": false,
        "updater.secret": "***REMOVED SENSITIVE VALUE***",
        "updater.release.channel": "stable",
        "overwrite.cli.url": "https:\/\/share.uri-international.de\/",
        "app_install_overwrite": [
            "calendar"
        ]
    }
}
or 

</details>

**Are you using external storage, if yes which one:** local/smb/sftp/...
None
**Are you using encryption:** yes/no
no
**Are you using an external user-backend, if yes which one:** LDAP/ActiveDirectory/Webdav/...
no
### Client configuration
**Browser:**
firefox 68
**Operating system:**
Ubuntu Linux 18.04 8(Linux Mint)
### Logs
#### Web server error log
<details>
<summary>Web server error log</summary>

Insert your webserver log here
none other than access logs here.

</details>

#### Nextcloud log (data/nextcloud.log)
<details>
<summary>Nextcloud log</summary>

Insert your Nextcloud log here

{"reqId":"XbTOxHo11ef4Xqp0teYOvQAAABI","level":3,"time":"2019-10-26T22:55:00+00:00","remoteAddr":"192.168.202.1","user":"--","app":"jsresourceloader","method":"GET","url":"\/index.php\/settings\/integrity\/failed","message":"Could not find resource js\/config.js to load","userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:69.0) Gecko\/20100101 Firefox\/69.0","version":"16.0.4.1"}
{"reqId":"XbTOxHo11ef4Xqp0teYOvQAAABI","level":3,"time":"2019-10-26T22:55:00+00:00","remoteAddr":"192.168.202.1","user":"--","app":"no app in context","method":"GET","url":"\/index.php\/settings\/integrity\/failed","message":{"Exception":"Symfony\\Component\\Routing\\Exception\\RouteNotFoundException","Message":"Unable to generate a URL for the named route \"theming.Theming.getImage\" as such route does not exist.","Code":0,"Trace":[{"file":"\/var\/www\/share.uri-international.de\/share\/lib\/private\/Route\/Router.php","line":337,"function":"generate","class":"Symfony\\Component\\Routing\\Generator\\UrlGenerator","type":"->","args":["theming.Theming.getImage",{"key":"logo","useSvg":true,"v":"10"},1]},{"file":"\/var\/www\/share.uri-international.de\/share\/lib\/private\/URLGenerator.php","line":79,"function":"generate","class":"OC\\Route\\Router","type":"->","args":["theming.Theming.getImage",{"key":"logo","useSvg":true,"v":"10"}]},{"file":"\/var\/www\/share.uri-international.de\/share\/apps\/theming\/lib\/ThemingDefaults.php","line":241,"function":"linkToRoute","class":"OC\\URLGenerator","type":"->","args":["theming.Theming.getImage",{"key":"logo","useSvg":true,"v":"10"}]},{"file":"\/var\/www\/share.uri-international.de\/share\/lib\/public\/Defaults.php","line":187,"function":"getLogo","class":"OCA\\Theming\\ThemingDefaults","type":"->","args":[true]},{"file":"\/var\/www\/share.uri-international.de\/share\/core\/templates\/layout.guest.php","line":42,"function":"getLogo","class":"OCP\\Defaults","type":"->","args":[]},{"file":"\/var\/www\/share.uri-international.de\/share\/lib\/private\/Template\/Base.php","line":178,"args":["\/var\/www\/share.uri-international.de\/share\/core\/templates\/layout.guest.php"],"function":"include"},{"file":"\/var\/www\/share.uri-international.de\/share\/lib\/private\/Template\/Base.php","line":150,"function":"load","class":"OC\\Template\\Base","type":"->","args":["\/var\/www\/share.uri-international.de\/share\/core\/templates\/layout.guest.php",null]},{"file":"\/var\/www\/share.uri-international.de\/share\/lib\/private\/legacy\/template.php","line":180,"function":"fetchPage","class":"OC\\Template\\Base","type":"->","args":[null]},{"file":"\/var\/www\/share.uri-international.de\/share\/lib\/private\/legacy\/template.php","line":211,"function":"fetchPage","class":"OC_Template","type":"->","args":[null]},{"file":"\/var\/www\/share.uri-international.de\/share\/lib\/private\/Template\/Base.php","line":131,"function":"fetchPage","class":"OC_Template","type":"->","args":[]},{"file":"\/var\/www\/share.uri-international.de\/share\/lib\/base.php","line":399,"function":"printPage","class":"OC\\Template\\Base","type":"->","args":[]},{"file":"\/var\/www\/share.uri-international.de\/share\/lib\/base.php","line":955,"function":"printUpgradePage","class":"OC","type":"::","args":[{"__class__":"OC\\SystemConfig"}]},{"file":"\/var\/www\/share.uri-international.de\/share\/index.php","line":42,"function":"handleRequest","class":"OC","type":"::","args":[]}],"File":"\/var\/www\/share.uri-international.de\/share\/3rdparty\/symfony\/routing\/Generator\/UrlGenerator.php","Line":130,"CustomMessage":"--"},"userAgent":"Mozilla\/5.0 (X11; Ubuntu; Linux x86_64; rv:69.0) Gecko\/20100101 Firefox\/69.0","version":"16.0.4.1"}
</details>

#### Browser log
<details>
<summary>Browser log</summary>

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...

</details>

#### Additional information:
With upgrade from 11 (unknown) to next step the calendar created errors already and it was ignored as have another parallel installation the worked fine so far (and upgraded from 16.0.4->17.0.0 fine.). Subsequent upgrades from this cloud worked fine for some time over versions 13,14,15 and 16 (with unknown minor versions). Calendar was not used and had problems , calendars could not be created.
@gaya4 gaya4 added 0. Needs triage Pending check for reproducibility or if it fits our roadmap bug labels Oct 26, 2019
@NilixMaster
Copy link

NilixMaster commented Oct 27, 2019

Hi,
the same with me as with gaya4.

Exception that I use MySql instead of PostgreSQL.

The attempt to add the field: "oc_calendartype" to the table fails because the field is deleted from the table every time I try to upgrade again.

Alter Table:
ALTER Table MyDataBase.oc_calendarobjects add column calendartype int(11) default 0 not null;
commit;

After changing the table, it is filled with the field and the value 0. After a new upgrade attempt, the field is removed from the table and runs on the above error.

@kesselb kesselb added 1. to develop Accepted and waiting to be taken care of and removed 0. Needs triage Pending check for reproducibility or if it fits our roadmap labels Oct 27, 2019
@kesselb

This comment has been minimized.

@tcitworld
Copy link
Member

tcitworld commented Oct 27, 2019

Odd, calendartype is definitely added by https://github.com/nextcloud/server/blob/stable16/apps/dav/lib/Migration/Version1006Date20180628111625.php#L59 in stable16 (and even 15!).
Isn't it related to #12410 ?

@kesselb
Copy link
Contributor

kesselb commented Oct 27, 2019

Odd, calendartype is definitely added by https://github.com/nextcloud/server/blob/stable16/apps/dav/lib/Migration/Version1006Date20180628111625.php#L59 in stable16 (and even 15!).
Isn't it related to #12410 ?

Ouch 🙈 c.calendartype => calendarobjects.calendartype

After changing the table, it is filled with the field and the value 0. After a new upgrade attempt, the field is removed from the table and runs on the above error.

Scary. Have not found any code that deletes this column 🤔

@gaya4
Copy link
Author

gaya4 commented Oct 27, 2019

mind, that in my opening comment I mentioned that I have 2 domains. The other updated fine. The faulty one only had an issue with the calendars in the past. Not sure that is related with NilixMaster.

@kesselb kesselb added 0. Needs triage Pending check for reproducibility or if it fits our roadmap and removed 1. to develop Accepted and waiting to be taken care of labels Oct 27, 2019
@andreas-p
Copy link

Just had the same problem upgrading from 15.x to 16.0.5 using the downoaded zip.
After rm -r apps core lib updater 3rdparty oc_ calendarobjects.calendartype didn't get removed any more, and the update went fine. So apparently some old left-over files were harmful.

@gaya4
Copy link
Author

gaya4 commented Nov 3, 2019

@andreas-p that worked well as it seems.
After deleting the files and then re-running the webapp-config -U updater the upgrade went through fine. For what it is worth:
Following apps have been disabled: calendar (incompatible), contacts (incompatible), files_frommail (incompatible)
Detailed logs
Preparing update
Set log level to debug
Turned on maintenance mode
Repair step: Repair MySQL collation
Repair info: Not a mysql database -> nothing to do
Repair step: Repair SQLite autoincrement
Repair step: Copy data from accounts table when migrating from ownCloud
Repair step: Drop account terms table when migrating from ownCloud
Updating database schema
Updated database
Repair step: Fix component of birthday calendars
Repair info: 3 birthday calendars updated.
Repair step: Regenerating birthday calendars to use new icons and fix old birthday events without year
Repair info: Repair step already executed
Repair step: Fix broken values of calendar objects
[0 / 0]: Fix broken values of calendar objects
Repair step: Registering building of calendar search index as background job
Repair info: Repair step already executed
Repair step: Registering background jobs to update cache for webcal calendars
Repair info: Added 0 background jobs to update webcal calendars
Repair step: Registering building of calendar reminder index as background job
Repair info: Repair step already executed
Repair step: Clean up orphan event and contact data
Repair info: 0 events without a calendar have been cleaned up
Repair info: 0 properties without an events have been cleaned up
Repair info: 0 changes without a calendar have been cleaned up
Repair info: 0 cached events without a calendar subscription have been cleaned up
Repair info: 0 changes without a calendar subscription have been cleaned up
Repair info: 0 contacts without an addressbook have been cleaned up
Repair info: 0 properties without a contact have been cleaned up
Repair info: 0 changes without an addressbook have been cleaned up
Repair step: Remove activity entries of private events
Repair info: Removed 0 activity entries
Updated "dav" to 1.13.0
Repair step: Fix the share type of guest shares when migrating from ownCloud
Repair step: Copy the share password into the dedicated column
Updated "files_sharing" to 1.9.0
Updated "files_trashbin" to 1.7.0
Updated "workflowengine" to 1.7.0
Updated "admin_audit" to 1.7.0
Updated "notifications" to 2.5.0
Updated "theming" to 1.8.0
Checking for update of app "accessibility" in appstore
Checked for update of app "accessibility" in appstore
Checking for update of app "activity" in appstore
Checked for update of app "activity" in appstore
Checking for update of app "admin_audit" in appstore
Checked for update of app "admin_audit" in appstore
Checking for update of app "cloud_federation_api" in appstore
Checked for update of app "cloud_federation_api" in appstore
Checking for update of app "dav" in appstore
Checked for update of app "dav" in appstore
Checking for update of app "federatedfilesharing" in appstore
Checked for update of app "federatedfilesharing" in appstore
Checking for update of app "files" in appstore
Checked for update of app "files" in appstore
Checking for update of app "files_pdfviewer" in appstore
Checked for update of app "files_pdfviewer" in appstore
Checking for update of app "files_rightclick" in appstore
Update app "files_rightclick" from appstore
Checked for update of app "files_rightclick" in appstore
Checking for update of app "files_sharing" in appstore
Checked for update of app "files_sharing" in appstore
Checking for update of app "files_trashbin" in appstore
Checked for update of app "files_trashbin" in appstore
Checking for update of app "files_videoplayer" in appstore
Checked for update of app "files_videoplayer" in appstore
Checking for update of app "gallery" in appstore
Checked for update of app "gallery" in appstore
Checking for update of app "lookup_server_connector" in appstore
Checked for update of app "lookup_server_connector" in appstore
Checking for update of app "notifications" in appstore
Checked for update of app "notifications" in appstore
Checking for update of app "oauth2" in appstore
Checked for update of app "oauth2" in appstore
Checking for update of app "provisioning_api" in appstore
Checked for update of app "provisioning_api" in appstore
Checking for update of app "theming" in appstore
Checked for update of app "theming" in appstore
Checking for update of app "twofactor_backupcodes" in appstore
Checked for update of app "twofactor_backupcodes" in appstore
Checking for update of app "updatenotification" in appstore
Checked for update of app "updatenotification" in appstore
Checking for update of app "viewer" in appstore
Checked for update of app "viewer" in appstore
Checking for update of app "workflowengine" in appstore
Checked for update of app "workflowengine" in appstore
Checking for update of app "calendar" in appstore
Checked for update of app "calendar" in appstore
Checking for update of app "contacts" in appstore
Checked for update of app "contacts" in appstore
Checking for update of app "files_frommail" in appstore
Checked for update of app "files_frommail" in appstore
Repair step: Repair MySQL collation
Repair info: Not a mysql database -> nothing to do
Repair step: Repair mime types
Repair step: Clean tags and favorites
Repair info: 0 tags of deleted users have been removed.
Repair info: 0 tags for delete files have been removed.
Repair info: 0 tag entries for deleted tags have been removed.
Repair info: 0 tags with no entries have been removed.
Repair step: Repair invalid shares
Repair step: Move .step file of updater to backup location
Repair step: Fix potential broken mount points
Repair info: No mounts updated
Repair step: Add log rotate job
Repair step: Clear frontend caches
Repair info: Image cache cleared
Repair info: SCSS cache cleared
Repair info: JS cache cleared
Repair step: Clear every generated avatar on major updates
Repair step: Add preview background cleanup job
Repair step: Queue a one-time job to cleanup old backups of the updater
Repair step: Cleanup invalid photocache files for carddav
Repair step: Add background job to cleanup login flow v2 tokens
Repair step: Remove potentially over exposing share links
Repair info: No need to remove link shares.
Repair step: Clear access cache of projects
Repair step: Cleanup cypress files from viewer app
Repair step: Switches from deprecated "production" to "stable" update channel
Repair step: Sets the enterprise logo
Starting code integrity check
Finished code integrity check
Turned off maintenance mode
Reset log level
Following apps have been disabled:
calendar (incompatible),
contacts (incompatible),
files_frommail (incompatible)

Now I'm logged in and indeed this even repaired my calendar on that cloud.
As per opening comment, the calendar didn't appear anymore and threw up errors on client (no calendar reaction on GUI) after upgrading somewhere about nextcloud 12+ (unknown). Now that very same calendar is visible again.
When checking the timeline for the nextcloud releases it must be Nextcloud 14 that made something with my calendars. @andreas-p (rm -r apps core lib updater 3rdparty) did fix that and the old calendar is accessible again.

The incompatible apps of calendar and contacts might have caused it? I am now logged in and can view the calendar without any particular calendar app installed, other than default.

Thank you.

Either this is worth improving the code or if you need something, I'll try to supply it. Or close it, if no need to work on it.

@girascal
Copy link

girascal commented Nov 4, 2019

After deleting the files and then re-running the webapp-config -U updater the upgrade went through fine. For what it is worth:

Hi I got the same issue while updating from 16.0.4 to 16.0.5 and I don't know how to resolve it
I saw that you where able to do it but I don't really understand the step you did so could you explain to me wich file you deleted so that I can get throught this update?

@ChipwizBen
Copy link

I also have the same issue with the column deleting itself after adding it in manually. This is an upgrade from 16.0.4 to 17.0.0.

I've had issues with various missing columns in the past that I've had to add manually to resolve problems like this, though this is the first where is hasn't worked (since it deletes it!). Is there a maintenance script that can be run to check each table for the correct columns and column types?

@ChipwizBen
Copy link

Also I keep seeing references to a migration process, e.g. (from the link above):
"This column was added in #10593, but the migration was never run"

What migration? Is there something that should be run before/after the upgrade process?

@nickvergessen
Copy link
Member

Is there something that should be run before/after the upgrade process?

No, it is part of the update process and should run automatically

@kesselb
Copy link
Contributor

kesselb commented Jan 20, 2020

We identified a possible root of those migration not executed issues: #18265

It's probably not the case for you but just to mention it. You are updating from Nextcloud 12. The update process is to extract the zip into the nextcloud folder. There are two way to make database changes: database.xml (old way), migrations (new way). Both ways are available. If a file $app/appinfo/database.xml exist migrations are skipped. Actually the integrity check should warn you about those files. Please make sure there is no apps/dav/appinfo/database.xml file.

@andreas-p
Copy link

This description sounds very plausible. It will happen when the previous directories weren't purged completely, but a manual unzip happened over the existing installation. The Integrity check will warn about this, but when it does it's already too late: occ upgrade was already executed.

@4001982248998
Copy link

4001982248998 commented Mar 6, 2020

also happened to me during update from 17 to 18

Doctrine\\DBAL\\Exception\\InvalidFieldNameException: An exception occurred while executing 'SELECT `c`.`id` FROM `oc_calendarchanges` `c` LEFT JOIN `oc_calendars` `p` ON `c`.`calendarid` = `p`.`id` WHERE (`p`.`id` IS NULL) AND (`c`.`calendartype` = ?)' with params [0]:
SQLSTATE[42S22]: Column not found: 1054 Unknown column 'c.calendartype' in 'where clause'

That was a “clean” install, not overwriting old version directories.

@4001982248998
Copy link

so, I think I got it working with these database manipulation steps:

ALTER TABLE oc_calendarobjects ADD COLUMN calendartype int(11) default 0 not null;
ALTER TABLE oc_calendarchanges ADD COLUMN calendartype int(11) default 0 not null;
ALTER TABLE oc_calendarsubscriptions ADD COLUMN synctoken VARCHAR(255);
ALTER TABLE oc_cards ADD COLUMN uid VARCHAR(255);
ALTER TABLE oc_calendarobjects_props ADD COLUMN calendartype INT(11) default 0 not null;

@kesselb
Copy link
Contributor

kesselb commented Nov 28, 2020

I guess the issue is solved for most of you?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0. Needs triage Pending check for reproducibility or if it fits our roadmap bug feature: install and update needs info
Projects
None yet
Development

No branches or pull requests

9 participants