XSDs are not resolved from the jar when referenced via HTTPS #1153

philwebb · 2020-05-15T01:03:17Z

Environment

Java 8.

Liquibase Version:
3.8

Liquibase Integration & Version: <Pick one: CLI, maven, gradle, spring boot, servlet, etc.>
Spring Boot

Database Vendor & Version:
N/A

Operating System Type & Version:
MacOS

Description

If a migration XML file refers to the liqubase XSD using https, the local jar packaged version is not used. That can cause issues if liquibase.org is down.

Steps To Reproduce

Use a liquibase XML with the following header without a network connection:

<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
        xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
        xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.8.xsd
        http://www.liquibase.org/xml/ns/dbchangelog-ext https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd">

Actual Behavior

XSD resolution fails because https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.8.xsd cannot be read.

Expected/Desired Behavior

The locally packaged XSD is used.

Additional Context

The StandardNamespaceDetails class could be updated to include https as well as http URLs.

The text was updated successfully, but these errors were encountered:

biswell · 2020-05-15T01:22:15Z

We are seeing a similar issue but with http as opposed to https when running form Maven.

Failed to read schema document 'http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.3.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not xsd:schema.

What is strange is we can open the XSDs in the browser. I have seen this in the past and the issue resolve itself so I guess it is network related.

gnumilanix · 2020-05-15T03:24:03Z

Same for us. Temporarily fixed by manually downloading those and storing it in s3.

fo-fo · 2020-05-15T05:34:55Z

Same issue here (or some variation of it, not related to https in our case). I traced it down to www.liquibase.org returning HTTP 403 if it doesn't like the user agent specified in headers.

curl -v -H "user-agent: Java/1.8.0_172" http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.6.xsd => 403

curl -v -H "user-agent: wtf" http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.6.xsd => 200

teamperfomatix · 2020-05-15T07:52:09Z

We are also facing the same issue in multiple Java projects.

Error setting up or running Liquibase: liquibase.exception.SetupException: Error parsing line 7 column 139 of config/liquibase/changelog/20180606073206_added_entity_Brand.xml: schema_reference.4: Failed to read schema document 'http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd', because

could not find the document;
the document could not be read;
the root element of the document is not xsd:schema.[m: Server returned HTTP response code: 403 for URL: http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd -> [1m[Help 1][m

bjsee · 2020-05-15T08:16:01Z

We are facing the same problem for our Java 8 systems today. Yesterday everything worked...
The problem seems to be with the user-agent header. When the value is "java/1.8.0_172" we get the 403 error. If I try with "java/11" it works (That's the reason why our Java 11 systems work properly).

tolix · 2020-05-15T09:34:13Z

i tried add to bat-launcher parameter -Dhttp.agent="wtf" (probably any except java/1.8*) and it helped :)

molivasdat · 2020-05-15T12:09:03Z

Hi @philwebb , @biswell @gnumilanix @fo-fo @teamperfomatix @bjsee @tolix . Thanks for reporting this. We migrated our website yesterday US CT and are working to fix this issues as quickly as we can to match the old site.

jkiljanski · 2020-05-15T12:53:47Z

Liquibase uses internal caching mechanism with all xsds up to it's version. If use use version 3.4 and you try to use version 3.5 of xsd, it will blow up because of the issue with agent described above. A quick fix is to download dbchangelog-3.5.xsd and put it on your classpath under:
xxx/src/main/resources/liquibase/parser/core/xml/dbchangelog-3.5.xsd
So liquibase will find it among other xsds.
When running it manaully using liquibase jar, remeber to add jar with xsd to a classpath.

molivasdat · 2020-05-15T13:39:12Z

Hi @philwebb , @biswell @gnumilanix @fo-fo @teamperfomatix @bjsee @tolix @jkiljanski . This issue should be resolved now. Thanks for the comments and workarounds!

nvoxland · 2020-05-15T13:48:23Z

Note: the issue that showed up yesterday with the http version related to the site hosting switch is unrelated to this issue with https URLs not being served from the local jar.

Liquibase has logic to use a copy of the xsd that is bundled with Liquibase, but that logic doesn't currently check for https urls, only http ones. Using http in your xsd references is a workaround for this issue.

Separately, there was a bug in a couple earlier versions of liquibase where http URLs were not being pulled from the local cache correctly, but that has been resolved a while ago. The website change would be affecting people using those older versions of liquibase only and that website issue should be resolved now.

gnumilanix · 2020-05-16T07:03:00Z

Hi @philwebb , @biswell @gnumilanix @fo-fo @teamperfomatix @bjsee @tolix @jkiljanski . This issue should be resolved now. Thanks for the comments and workarounds!

Verified and it's resolved. Thank you

molivasdat · 2020-05-20T05:23:59Z

Verified that this has been resolved

wilx · 2021-04-21T11:13:42Z

Can you share how was this resolved? Which version has this fixed? I am seeing similar symptoms with 3.8.9.

molivasdat · 2021-04-21T12:42:40Z

Back in May of last year, we updated our website and moved providers. When that happened, the urls to the xsds were incorrect and quickly fixed.

Quiark · 2022-01-07T06:18:06Z

The problem is back today.

JamesBarnes88 · 2022-01-07T07:31:02Z

Seeing this issue has come back again.

kdkmadusanka · 2022-01-07T07:52:41Z

Issue came back again

sureshkumarn86 · 2022-01-07T08:06:03Z

facing issue today

JWiddershoven · 2022-01-07T08:07:38Z

Seeing the same issue with HTTP:
Failed to read schema document 'http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.6.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.

JamesBarnes88 · 2022-01-07T08:25:50Z

Was able to get this working by making sure the liquibase-core pom version and the xsd version match. As someone mentioned, there is a fallback to use the bundled xsd files. In my case, was looking for 3.6 xsd and my pom version was 3.5. 😞

wanghengxn · 2022-01-07T08:52:42Z

I have solved the problem.Ensure that the xsd version in the master.xml and changelog.xml are the same. The liquibase-core version maybe is not the point in pom.

MaDDogSGP · 2022-01-07T09:18:53Z

Folks, Jono from Singapore. My colleagues is experiencing road blocks in resolving the issue. Any kind folks can advise,, please?

MaDDogSGP · 2022-01-07T09:23:23Z

There seems to be a LiquiBase Validation Issue when deploying to Production. Any ideas?

SirWojtek · 2022-01-07T09:27:57Z

We use v3.6 and updating the URL to the XSD file to point to the corresponding version of the schema does not help. Tried both HTTP and HTTPS.

albert-stec · 2022-01-07T09:31:57Z

Same for v3.3.5 not working. Updating to 3.5.1 helped.

SirWojtek · 2022-01-07T09:37:29Z

Hate to bother you @molivasdat, could it be related to some sort of a hosting issue as before?

kdkmadusanka · 2022-01-07T10:24:23Z

after updating file versions we are getting validation failures from liquibase. Anyone got validation failures..?

sergts · 2022-01-07T11:08:26Z

Please be very careful when changing liquibase-core version. Changing it from 3.5.5 to 3.6.1 solved the problem for 3.6 xsd, however, WHEN THE LIQUIBASE WAS RUN FOR THE FIRST TIME AFTER VERSION CHANGE IT DID NOT SEE PREVIOUS CHANGESETS! It ran as if dbchangelog was empty. Luckily our first changeset tried to create an already existing table, thus process failed and on second rerun it saw dbchangelog normally. But if you have drop statements in your first changeset, you may lose data.

Edit: if you can't change version for this reason, then you can try

change http to https in changelog.xml and
add JVM option -Dhttp.agent="workaround"

andrei-tulba · 2022-01-07T12:41:20Z

The same issue I started to get today but the folowing code solved my problem

Before:

 new Liquibase(
              changeLog.getPath(),
                  new FileSystemResourceAccessor(new File("."))),
              new JdbcConnection(connection)).update(new Contexts());

After:

new Liquibase(
              changeLog.getPath(),
              new CompositeResourceAccessor(
                  new FileSystemResourceAccessor(new File(".")), new ClassLoaderResourceAccessor()),
              new JdbcConnection(connection).update(new Contexts());

molivasdat · 2022-01-07T13:47:02Z

Hi @SirWojtek There was a website change last night. We are looking into it.

molivasdat · 2022-01-07T14:29:22Z

Hi team. The issue should be resolved now.

molivasdat · 2022-01-07T14:33:12Z

Hi team. The issue should be resolved now.
@Quiark @JamesBarnes88 @kdkmadusanka @sureshkumarn86 @JWiddershoven @wanghengxn @MaDDogSGP @SirWojtek @albert-stec @sergts @andrei-tulba Both http and https should resolve now.
This time it was http received a 301 and https resolved just fine. We have fixed the issue with our website.

gzalo · 2022-01-07T14:46:46Z

@molivasdat Still seeing the issue:

$ curl  -H "user-agent: Java/1.8.0_172" https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx</center>
</body>
</html>

(but works just fine if not using that user agent)

bschlosser · 2022-01-07T14:49:53Z

@molivasdat I'm seeing the 403 using the latest OpenJdk Java 8 1.8.0_312-b07 build as well

Using the workaround of adding -Dhttp.agent="workaround" still works.

molivasdat · 2022-01-07T14:57:23Z

Hi @gzalo Thanks for the update. We will look at it.

setu9760 · 2022-01-07T15:23:04Z

We are still seeing this issue for this url 'https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-1.9.xsd'.

While we are able to workaround by updating the xsd version or changing to http protocol it still is a blocker in certain scenarios where the liquibase xmls are packaged into previously built jar files.

below is the output for curl command when user-agent is Java. any other random value works

curl -v --head -H "user-agent: Java/1.8.0_275" https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-1.9.xsd
*   Trying 35.224.204.50...
* TCP_NODELAY set
* Connected to www.liquibase.org (35.224.204.50) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=www.liquibase.org
*  start date: Nov  9 21:23:12 2021 GMT
*  expire date: Feb  7 21:23:11 2022 GMT
*  subjectAltName: host "www.liquibase.org" matched cert's "www.liquibase.org"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fd4f6808600)
> HEAD /xml/ns/dbchangelog/dbchangelog-1.9.xsd HTTP/2
> Host: www.liquibase.org
> Accept: */*
> user-agent: Java/1.8.0_275
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 403
HTTP/2 403
< server: nginx
server: nginx
< date: Fri, 07 Jan 2022 15:20:46 GMT
date: Fri, 07 Jan 2022 15:20:46 GMT
< content-type: text/html
content-type: text/html
< content-length: 146
content-length: 146
< vary: Accept-Encoding
vary: Accept-Encoding

<
* Connection #0 to host www.liquibase.org left intact
* Closing connection 0

nvoxland · 2022-01-07T15:39:42Z

We're working on the java user agent setting. You can follow along at #2342

molivasdat · 2022-01-07T16:51:21Z

@gzalo @bschlosser @setu9760, we have now fixed the issue.

molivasdat · 2022-01-07T17:02:20Z

ISSUE RESOLVED: During a regular website maintenance, a change was introduced that incorrectly forced https for the xsd files we host for some Liquibase users, causing some users to experience 301 and 403 errors. This issue has been resolved and we apologize for any inconvenience.

zambrovski · 2022-02-01T08:41:54Z

Hi folks,

the error re-appeared again. Today, the call curl -v http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.9.xsd delivers 301 moved permanently (Location: https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.9.xsd).

SAX Parsers in Keycloak for example are not able to get them. Could you please re-enable the unsafe http access?

Thanks,

Simon

polarfish · 2022-02-09T10:30:21Z

The original issue is still reproducible.

Java 11.

Liquibase Version:
3.10.3

Liquibase Integration & Version: <Pick one: CLI, maven, gradle, spring boot, servlet, etc.>
Spring Boot 2.5.1

Database Vendor & Version:
N/A

Operating System Type & Version:
MacOS

When I start my service offline I have the behaviour below

https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.8.xsd -> FAIL
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.8.xsd -> OK

https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.10.xsd -> FAIL
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.10.xsd -> OK

https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.5.xsd -> FAIL
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.5.xsd -> FAIL

tomunix2000 · 2023-02-19T06:52:49Z

Problem is still there with 4.2.0, both http and https
[ERROR] Error setting up or running Liquibase:[ERROR] Error parsing line 10 column 110 of src/main/resources/changelog.xml: schema_reference.4: Failed to read schema document 'https://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-latest.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not xsd:schema.

and

[ERROR] Error setting up or running Liquibase:[ERROR] Error parsing line 7 column 103 of src/main/resources/changelog.xml: schema_reference.4: Failed to read schema document 'http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-4.4.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not xsd:schema.

molivasdat closed this as completed May 20, 2020

This was referenced Feb 1, 2022

Restore HTTP access to liquibase schemas #2449

Closed

Provide a local XSD file for liquibase schema Captain-P-Goldfish/scim-for-keycloak#44

Closed

polarfish mentioned this issue Feb 11, 2022

XSDs are not resolved from the jar when referenced via HTTPS (liquibase 3.x) #2520

Closed

anandjakhaniya mentioned this issue Apr 13, 2022

Fix liquibase trying to get the xsd from the internet VertaAI/modeldb#2997

Merged

XSDs are not resolved from the jar when referenced via HTTPS #1153

XSDs are not resolved from the jar when referenced via HTTPS #1153

Comments

philwebb commented May 15, 2020 • edited by sync-by-unito bot Loading

Environment

Description

Steps To Reproduce

Actual Behavior

Expected/Desired Behavior

Additional Context

biswell commented May 15, 2020

gnumilanix commented May 15, 2020

fo-fo commented May 15, 2020

teamperfomatix commented May 15, 2020 • edited Loading

bjsee commented May 15, 2020

tolix commented May 15, 2020

molivasdat commented May 15, 2020

jkiljanski commented May 15, 2020 • edited Loading

molivasdat commented May 15, 2020

nvoxland commented May 15, 2020 • edited Loading

gnumilanix commented May 16, 2020

molivasdat commented May 20, 2020

wilx commented Apr 21, 2021

molivasdat commented Apr 21, 2021

Quiark commented Jan 7, 2022

JamesBarnes88 commented Jan 7, 2022

kdkmadusanka commented Jan 7, 2022

sureshkumarn86 commented Jan 7, 2022

JWiddershoven commented Jan 7, 2022

JamesBarnes88 commented Jan 7, 2022

wanghengxn commented Jan 7, 2022

MaDDogSGP commented Jan 7, 2022

MaDDogSGP commented Jan 7, 2022

SirWojtek commented Jan 7, 2022 • edited Loading

albert-stec commented Jan 7, 2022

SirWojtek commented Jan 7, 2022

kdkmadusanka commented Jan 7, 2022

sergts commented Jan 7, 2022 • edited Loading

andrei-tulba commented Jan 7, 2022

molivasdat commented Jan 7, 2022

molivasdat commented Jan 7, 2022

molivasdat commented Jan 7, 2022

gzalo commented Jan 7, 2022 • edited Loading

bschlosser commented Jan 7, 2022

molivasdat commented Jan 7, 2022

setu9760 commented Jan 7, 2022

nvoxland commented Jan 7, 2022

molivasdat commented Jan 7, 2022

molivasdat commented Jan 7, 2022

zambrovski commented Feb 1, 2022

polarfish commented Feb 9, 2022 • edited Loading

tomunix2000 commented Feb 19, 2023

philwebb commented May 15, 2020 •

edited by sync-by-unito bot

Loading

teamperfomatix commented May 15, 2020 •

edited

Loading

jkiljanski commented May 15, 2020 •

edited

Loading

nvoxland commented May 15, 2020 •

edited

Loading

SirWojtek commented Jan 7, 2022 •

edited

Loading

sergts commented Jan 7, 2022 •

edited

Loading

gzalo commented Jan 7, 2022 •

edited

Loading

polarfish commented Feb 9, 2022 •

edited

Loading