[HTTPS] Finalize update server to https process #9579

Merged
merged 3 commits into from Apr 15, 2016

Projects

None yet

10 participants

@andrepereiradasilva
Contributor

Summary of Changes

This PR is a revival of #8196 that was closed because by then the Joomla update server did not fully supported HTTPS.

Now that the update server is working in HTTPS i think we can now safely update all update.joomla.org http:// references to https://.

Testing Instructions

Experience joomla developers can check the code review.
For more extensive tests:

Update
  1. Apply this patch to latest staging or 3.5.0
  2. Run the queries from https://raw.githubusercontent.com/andrepereiradasilva/joomla-cms/update-https/administrator/components/com_admin/sql/updates/mysql/3.5.2-2016-04-01.sql (replace #__ for your table prefix)
  3. Check if the update server URI are in https in #__update_sites table.
  4. Test if fetching for a joomla update does not give any error.
New installation
  1. Download patched Joomla 3.5.0 (https://github.com/andrepereiradasilva/joomla-cms/archive/update-https.zip)
  2. Install joomla
  3. Check if the update server URI are in https in #__update_sites table.
  4. Test if fetching for a joomla update does not give any error.
@andrepereiradasilva andrepereiradasilva update server to https
40d4ef4
@dgt41
Member
dgt41 commented Mar 25, 2016

@andrepereiradasilva instead of asking ppl to patch themselves the 3.5 you can ask them to use your repo: https://github.com/andrepereiradasilva/joomla-cms/archive/update-https.zip

@andrepereiradasilva
Contributor

updated thanks @dgt41

@waader
Contributor
waader commented Apr 1, 2016

I have tested this item successfully on 40d4ef4


This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/9579.

@wojsmol
Contributor
wojsmol commented Apr 1, 2016

@andrepereiradasilva Please rebase this PR

@andrepereiradasilva
Contributor

yes, will do when i have time.

@wojsmol wojsmol referenced this pull request in andrepereiradasilva/joomla-cms Apr 1, 2016
Closed

rebase 3.5.1-2016-03-25.sql #25

@wojsmol wojsmol referenced this pull request in andrepereiradasilva/joomla-cms Apr 1, 2016
Merged

redo of #25 #26

@andrepereiradasilva andrepereiradasilva fix conflicts 83f1661
@andrepereiradasilva andrepereiradasilva add scripts to 3.5.2
8f9dcd6
@joomla-cms-bot

This PR has received new commits.

CC: @waader


This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/9579.

@andrepereiradasilva
Contributor

just fixed conflicts no other change

@brianteeman
Contributor

I have tested this item successfully on 8f9dcd6


This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/9579.

@andrepereiradasilva
Contributor

@waader can you retest after the latest (solve conflicts) changes?

@conconnl

I have tested this item successfully on 8f9dcd6

Tested Find Updates after clearing cache with UpdateSite too Joomla! updates on HTTPS.


This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/9579.

@brianteeman
Contributor

RTC


This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/9579.

@joomla-cms-bot joomla-cms-bot added the RTC label Apr 15, 2016
@brianteeman
Contributor

@rdeutz your choice on the milestone ;)


This comment was created with the J!Tracker Application at issues.joomla.org/joomla-cms/9579.

@rdeutz rdeutz added this to the Joomla 3.5.2 milestone Apr 15, 2016
@rdeutz rdeutz merged commit b6896eb into joomla:staging Apr 15, 2016

2 checks passed

JTracker/HumanTestResults Human Test Results: 2 Successful 0 Failed.
Details
continuous-integration/travis-ci/pr The Travis CI build passed
Details
@joomla-cms-bot joomla-cms-bot removed the RTC label Apr 15, 2016
@andrepereiradasilva andrepereiradasilva deleted the andrepereiradasilva:update-https branch Apr 15, 2016
@richard67
Contributor

@andrepereiradasilva Just for info: The changes in your update scripts "3.5.2-2016-04-01.sql" will be applied when people update with the Joomla! Update component (including new upload and install feature), because in this case the update scripts are just executed.

But when people update with the "copy the files or unzip the container and then run the db fixer" method, the statements will not be applied, because silly db fix only handles DDL statements (data definition language, "CREATE TABLE" or "ALTER TABLE"), but not DML (data manipulation language, "INSERT", "UPDATE", "DELETE").

Just in case you were not aware of this and later find this during testing somehow.

@andrepereiradasilva
Contributor
andrepereiradasilva commented Apr 15, 2016 edited

huumm i tought the db fixer would run all sql statements.
so there are a lot of sql in the sql updates dir that are not applied!

@richard67
Contributor

Yes, all insert update delete is not applied by the db fixer. And in the way things work now, it would not be possible to implement that in a safe way, using check queries for verifying if a statement has been applied, because some of the statements are executed when doing a discovery install, e.g. all those where new core components were introduced and so the schema updates contain inserts into the extensions table.

One way do solve that (with a certain risk) would be to do in the db fixer what the updater does, too: Run all schema update scripts with a version in their filename greater than the schema version saved in db. But this works only 1 time, because at the end, the schema manager sets the db schema to the version of the last file in the array, regardless if the file has been executed with success or if there were some errors.

Maybe there were some attempts to do such a correction in past, but people noticed it did not work?

@wilsonge could maybe know that.

I currently work a bit on the next step for utf8mb4: make it work for extensions, too, and use also versioned files like for schema updates, to when some new stuff to be added it does not require to run the complete conversion again.

When this will be done in a few days or some week or 2, my idea was to work on the db fixer to make it work better. Am not sure if I will still have enought time then, but hope so.

And maybe make it also handle extensions, but this could collide with what discovery install does, so am not sure about that thing.

@wojsmol
Contributor
wojsmol commented Apr 15, 2016 edited

@richard67 The situation described above is associated with the query syntax -See #8788. Already working on PR.

@richard67
Contributor
richard67 commented Apr 15, 2016 edited

@wojsmol Did you comment on the wrong discussion?

What I discuss here with @andrepereiradasilva has absolutely nothing to do with the issue you refer to.

The issue you refer to has nothing to do with the fact that when you use the "copy the files or unzip the zip and then use the db fixer" method to update Joomla!, the schema manager (db fixer) does not apply INSERT or UPDATE or DELETE statements.

@wojsmol
Contributor
wojsmol commented Apr 15, 2016 edited

@richard67 No, see #8788 (comment) UPDATE vs UPDATE COLUMN.

@richard67
Contributor

@wojsmol Where in this sql handled there can you see any "UPDATE"? You are wrong! What @wilsonge meant in his comment is that the quoting of the column name is required for the schema manager to correctly detect the column name in an alter table statement, and yes, that's right. But it has nothing to do with the fact that the schema manager does not apply insert or update or delete statements when doing the database fix! @wilsonge Please confirm so he believes me.

@richard67
Contributor

Ahh I just see: The code link of @wilsonge in his comment does not fit anymore because the file meanwhile changed!!! He linked to line 25 once but meanwhile new code has been inserted in this file.

But again, it has nothing to do with this issue here, it only corrected a syntax error.

What I discuss here with @andrepereiradasilva is the fact that the schema manager not runs the schema update script completely when doing a db fix, it only checks and if necessary repairs statements "alter table" and "create table". I added a special thing for resetting the utf8mb4 conversion status, and this is now what is shown in line 25, but when George commented that, my code was not there yet.

So again: Beside this special update statement for this special column of this special table, the schema manager does not apply DML statements. Full stop.

@wilsonge again if you reead please confirm so I do not get nothered with this silly discussion!

@wojsmol
Contributor
wojsmol commented Apr 15, 2016 edited

@richard67 you are right, this is update for column content not definition as in #8788. Sorry for the confusion.

@richard67
Contributor
richard67 commented Apr 15, 2016 edited

@andrepereiradasilva The question remains what we can do to make your change from this PR here (and others with updates of database records e.g. for https) also be applied when people use the "copy files and then use db fixer" method.

That was the reason why I commented that here, because your changes are important, and those people will miss them.

It is a big mess what we have now in Joomla! with schema updating.

If you have any ideas or opinions let me know.

I wish @wilsonge or some other maintainer being familiar with the schema updating and db fixer would find the time for a discussion, because I have a few ideas which ways we could go to solve that, but I have no idea if some of these ideas once in past already have been tried by someone else and have been given up for some reason.

Unfortunately I do not plan to join JAB so we cannot do this discussion there, at least not with me.

But there has to be done something.

Or Joomla! stops to support and advertise this trouble causing update method, now as we have the "Upload & Install" option in the update component.

@infograf768
Member

Folks, I can't update or install languages anymore.
Is it related to this PR?

@infograf768
Member

Sorry, forget it: I had left the tmp folder readonly after some other tests. 😃

@brianteeman
Contributor

lol

@rdeutz rdeutz modified the milestone: Joomla 3.5.2, Joomla! 3.6.0 May 1, 2016
@wojsmol wojsmol referenced this pull request May 24, 2016
Merged

Rename update site names #10612

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment