ERROR: Unable to calculate checksum of <<inputfile.sql>> 4/9/2020 4:11:38 PMInput length = 1

[Sourav1407]

Which version and edition of Flyway are you using?

Flyway 6.3.3

If this is not the latest version, can you reproduce the issue with the latest one as well(Many bugs are fixed in newer releases and upgrading will often resolve the issue)

This issue was not there in 4.2.0 but when we tried to upgrade to 6.2.2 this issue appeared. Upgraded with the latest version i.e. 6.3.3 but the issue still persists

Which client are you using? (Command-line, Java API, Maven plugin, Gradle plugin)

Command-line

Which database are you using (type & version)?

PostgreSQL 11.6

Which operating system are you using?

RHEL 7.5

What did you do?

Upgraded the docker image which has flyway bundle as the migration tool. Image is having SQL files which are checked in by our engineering team members. They uses different types of editor such as pgadmin,dbeaver,notepad++

All was good in 4.2.0 until we decided to upgrade to 6.2.2 to fix a security issue which popped up due to 2 of the DB drivers present in the flyway db. Once we upgraded, the security issue was resolved as the 6.2.2 version of flyway was using latest versions of the drivers but we started facing this "unable to calculate checksum" issue. Then we had to downgrade it back to 4.2.0 and decided to remove the drivers from the pack which were causing the security issues. Removal of the drivers from the image solved the problem but we want to upgrade the flyway now so that we don't have to remove the drivers

What did you expect to see?

The expectation is that this issue should not pop up as we can't go and correct the type conversion to UTF-8 for each and every file. For lot of files used for different engines and modules we are facing this. Even we tried to convert one problematic file to UTF-8 in notepad++ but we were still receiving the same error. We have lot of SQL files checked in for last few years and now they have grown up so much that we can't go and correct each of them or ask our engineers to correct them and also we don't even know how the file changes will impact the upgrade process in customer environments using our product and also as I said we are not sure also that the issue will be resolved post the changes of UTF-8 conversion. Any configuration tweak may also help if there is a way so that this error can be bypassed.

What did you see instead?

The expectation is that the issue should not pop up and the latest version of flyway and it should behave like 4.2.0 so that we can upgrade the latest flyway db in our database image

[MikielAgutu]

Can you share the file that is causing the issue?

[Sourav1407]

@MikielAgutu We are speaking with our legal team on sharing the file. I will get back to you

[juliahayward]

What character encoding are you using? Is it consistent across your migration files?

[MikielAgutu]

@Sourav1407 Any news?

[Sourav1407]

@juliahayward Most of them are UTF-8. We thought the problematic file might be having a different encoding and we tried to even convert that to UTF-8 in notepad++ but still the issue was not resolved.
But the concern is that while using 4.2.0 we never faced such issue and it worked like a charm

[Sourav1407]

@MikielAgutu Awaiting response from our legal team. As soon as we get a reply will share the file. Thanks for your support on the same

[Sourav1407]

@MikielAgutu PFB one of the problematic file. I have modified the format to .txt from .sql
DB_sql_V3_20190813_27__Portuguese_Generic_Utterances.txt

[Sourav1407]

@MikielAgutu Tried with the latest one 6.4.4. Got similar errors for the same files

6/12/2020 12:34:31 PMFlyway Community Edition 6.4.4 by Redgate
6/12/2020 12:34:31 PMDatabase: jdbc:postgresql://postgreslb.postgresql:6432/cri (PostgreSQL 11.7)
6/12/2020 12:34:31 PMERROR: Unable to calculate checksum of V2_20171108_2__ActionScript.sql
6/12/2020 12:34:31 PMInput length = 1
6/12/2020 12:34:31 PMCaused by: java.nio.charset.MalformedInputException: Input length = 1

6/12/2020 12:34:31 PMFlyway Community Edition 6.4.4 by Redgate
6/12/2020 12:34:31 PMDatabase: jdbc:postgresql://postgreslb.postgresql:6432/cri (PostgreSQL 11.7)
6/12/2020 12:34:37 PMERROR: Unable to calculate checksum of V3_20190813_27__Portuguese_Generic_Utterances.sql
6/12/2020 12:34:37 PMInput length = 1
6/12/2020 12:34:37 PMCaused by: java.nio.charset.MalformedInputException: Input length = 1

[juliahayward]

It looks like an issue with the file encoding. If you set in the configuration file:

flyway.encoding=ISO_8859_1

does that make the migrations run?

[Sourav1407]

@juliahayward Getting checksum mismatch now after setting flyway.encoding=ISO_8859_1 for almost 100+ files. If I go and ask my deployment team to run the checksum repair in customer's production environment then lot of questions will be raised

6/12/2020 7:51:55 PMERROR: Validate failed:
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 1.20170802.1
6/12/2020 7:51:55 PM-> Applied to database : -1527526515
6/12/2020 7:51:55 PM-> Resolved locally : 1470590892
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20171108.2
6/12/2020 7:51:55 PM-> Applied to database : 102437194
6/12/2020 7:51:55 PM-> Resolved locally : -635328017
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20171121.8
6/12/2020 7:51:55 PM-> Applied to database : 529062811
6/12/2020 7:51:55 PM-> Resolved locally : -844667126
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20171121.9
6/12/2020 7:51:55 PM-> Applied to database : -1773929542
6/12/2020 7:51:55 PM-> Resolved locally : 2028432990
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20171121.12
6/12/2020 7:51:55 PM-> Applied to database : -92622635
6/12/2020 7:51:55 PM-> Resolved locally : -260531534
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20171121.14
6/12/2020 7:51:55 PM-> Applied to database : 1440963269
6/12/2020 7:51:55 PM-> Resolved locally : -290451099
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20171121.15
6/12/2020 7:51:55 PM-> Applied to database : -1435124649
6/12/2020 7:51:55 PM-> Resolved locally : 693197688
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20180103.1
6/12/2020 7:51:55 PM-> Applied to database : 1782171639
6/12/2020 7:51:55 PM-> Resolved locally : 2094907535
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20180103.2
6/12/2020 7:51:55 PM-> Applied to database : 1038704292
6/12/2020 7:51:55 PM-> Resolved locally : 1626189996
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20180103.5
6/12/2020 7:51:55 PM-> Applied to database : -757105328
6/12/2020 7:51:55 PM-> Resolved locally : 1831763030
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20180105.6
6/12/2020 7:51:55 PM-> Applied to database : -1745480510
6/12/2020 7:51:55 PM-> Resolved locally : -1638382870
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20180105.10
6/12/2020 7:51:55 PM-> Applied to database : 1040883703
6/12/2020 7:51:55 PM-> Resolved locally : -1249201146

[Sourav1407]

@juliahayward Got this for ~100+ files. Was not getting this for 4.2.0. Even if I repair then I don't know the impact and how this encoding is going to impact the future migrations. If I repair the checksum the checksum value will change and if I see any issues with future migrations I will not be able to even rollback to 4.2.0

[juliahayward]

OK, then remove that encoding setting... we will need to expand the encoding functionality so that you can override it on a per-script basis rather than all or nothing. I'll try to implement that in the next patch version, which should be out next week.

[Sourav1407]

@juliahayward Sure Thanks

[Sourav1407]

@juliahayward @Lyeeedar Closing comments please. If I upgrade to latest version, will the issue be resolved? Any encoding I have to provide in config?

[juliahayward]

Yes. You will need to add a file DB_sql_V3_20190813_27__Portuguese_Generic_Utterances.sql.conf next to the relevant migration that contains simply:

encoding=ISO_8859_1

and then all should be good. You don't need to alter any previous scripts and the encoding change is not required in the main configuration.