Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ERROR: Unable to calculate checksum of <<inputfile.sql>> 4/9/2020 4:11:38 PMInput length = 1 #2766

Closed
Sourav1407 opened this issue Apr 9, 2020 · 15 comments

Comments

@Sourav1407
Copy link

@Sourav1407 Sourav1407 commented Apr 9, 2020

Which version and edition of Flyway are you using?

Flyway 6.3.3

If this is not the latest version, can you reproduce the issue with the latest one as well(Many bugs are fixed in newer releases and upgrading will often resolve the issue)

This issue was not there in 4.2.0 but when we tried to upgrade to 6.2.2 this issue appeared. Upgraded with the latest version i.e. 6.3.3 but the issue still persists

Which client are you using? (Command-line, Java API, Maven plugin, Gradle plugin)

Command-line

Which database are you using (type & version)?

PostgreSQL 11.6

Which operating system are you using?

RHEL 7.5

What did you do?

Upgraded the docker image which has flyway bundle as the migration tool. Image is having SQL files which are checked in by our engineering team members. They uses different types of editor such as pgadmin,dbeaver,notepad++

All was good in 4.2.0 until we decided to upgrade to 6.2.2 to fix a security issue which popped up due to 2 of the DB drivers present in the flyway db. Once we upgraded, the security issue was resolved as the 6.2.2 version of flyway was using latest versions of the drivers but we started facing this "unable to calculate checksum" issue. Then we had to downgrade it back to 4.2.0 and decided to remove the drivers from the pack which were causing the security issues. Removal of the drivers from the image solved the problem but we want to upgrade the flyway now so that we don't have to remove the drivers

What did you expect to see?

The expectation is that this issue should not pop up as we can't go and correct the type conversion to UTF-8 for each and every file. For lot of files used for different engines and modules we are facing this. Even we tried to convert one problematic file to UTF-8 in notepad++ but we were still receiving the same error. We have lot of SQL files checked in for last few years and now they have grown up so much that we can't go and correct each of them or ask our engineers to correct them and also we don't even know how the file changes will impact the upgrade process in customer environments using our product and also as I said we are not sure also that the issue will be resolved post the changes of UTF-8 conversion. Any configuration tweak may also help if there is a way so that this error can be bypassed.

What did you see instead?

The expectation is that the issue should not pop up and the latest version of flyway and it should behave like 4.2.0 so that we can upgrade the latest flyway db in our database image

@MikielAgutu
Copy link
Contributor

@MikielAgutu MikielAgutu commented Apr 9, 2020

Can you share the file that is causing the issue?

@Sourav1407
Copy link
Author

@Sourav1407 Sourav1407 commented Apr 22, 2020

@MikielAgutu We are speaking with our legal team on sharing the file. I will get back to you

@MikielAgutu MikielAgutu added this to the On the Radar milestone Apr 22, 2020
@juliahayward
Copy link
Member

@juliahayward juliahayward commented Apr 29, 2020

What character encoding are you using? Is it consistent across your migration files?

@MikielAgutu
Copy link
Contributor

@MikielAgutu MikielAgutu commented May 28, 2020

@Sourav1407 Any news?

@Sourav1407
Copy link
Author

@Sourav1407 Sourav1407 commented Jun 1, 2020

@juliahayward Most of them are UTF-8. We thought the problematic file might be having a different encoding and we tried to even convert that to UTF-8 in notepad++ but still the issue was not resolved.
But the concern is that while using 4.2.0 we never faced such issue and it worked like a charm

@Sourav1407
Copy link
Author

@Sourav1407 Sourav1407 commented Jun 1, 2020

@MikielAgutu Awaiting response from our legal team. As soon as we get a reply will share the file. Thanks for your support on the same

@Sourav1407
Copy link
Author

@Sourav1407 Sourav1407 commented Jun 11, 2020

@MikielAgutu PFB one of the problematic file. I have modified the format to .txt from .sql
DB_sql_V3_20190813_27__Portuguese_Generic_Utterances.txt

@Sourav1407
Copy link
Author

@Sourav1407 Sourav1407 commented Jun 12, 2020

@MikielAgutu Tried with the latest one 6.4.4. Got similar errors for the same files

6/12/2020 12:34:31 PMFlyway Community Edition 6.4.4 by Redgate
6/12/2020 12:34:31 PMDatabase: jdbc:postgresql://postgreslb.postgresql:6432/cri (PostgreSQL 11.7)
6/12/2020 12:34:31 PMERROR: Unable to calculate checksum of V2_20171108_2__ActionScript.sql
6/12/2020 12:34:31 PMInput length = 1
6/12/2020 12:34:31 PMCaused by: java.nio.charset.MalformedInputException: Input length = 1

6/12/2020 12:34:31 PMFlyway Community Edition 6.4.4 by Redgate
6/12/2020 12:34:31 PMDatabase: jdbc:postgresql://postgreslb.postgresql:6432/cri (PostgreSQL 11.7)
6/12/2020 12:34:37 PMERROR: Unable to calculate checksum of V3_20190813_27__Portuguese_Generic_Utterances.sql
6/12/2020 12:34:37 PMInput length = 1
6/12/2020 12:34:37 PMCaused by: java.nio.charset.MalformedInputException: Input length = 1

@juliahayward
Copy link
Member

@juliahayward juliahayward commented Jun 12, 2020

It looks like an issue with the file encoding. If you set in the configuration file:

flyway.encoding=ISO_8859_1

does that make the migrations run?

@Sourav1407
Copy link
Author

@Sourav1407 Sourav1407 commented Jun 12, 2020

@juliahayward Getting checksum mismatch now after setting flyway.encoding=ISO_8859_1 for almost 100+ files. If I go and ask my deployment team to run the checksum repair in customer's production environment then lot of questions will be raised

6/12/2020 7:51:55 PMERROR: Validate failed:
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 1.20170802.1
6/12/2020 7:51:55 PM-> Applied to database : -1527526515
6/12/2020 7:51:55 PM-> Resolved locally : 1470590892
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20171108.2
6/12/2020 7:51:55 PM-> Applied to database : 102437194
6/12/2020 7:51:55 PM-> Resolved locally : -635328017
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20171121.8
6/12/2020 7:51:55 PM-> Applied to database : 529062811
6/12/2020 7:51:55 PM-> Resolved locally : -844667126
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20171121.9
6/12/2020 7:51:55 PM-> Applied to database : -1773929542
6/12/2020 7:51:55 PM-> Resolved locally : 2028432990
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20171121.12
6/12/2020 7:51:55 PM-> Applied to database : -92622635
6/12/2020 7:51:55 PM-> Resolved locally : -260531534
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20171121.14
6/12/2020 7:51:55 PM-> Applied to database : 1440963269
6/12/2020 7:51:55 PM-> Resolved locally : -290451099
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20171121.15
6/12/2020 7:51:55 PM-> Applied to database : -1435124649
6/12/2020 7:51:55 PM-> Resolved locally : 693197688
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20180103.1
6/12/2020 7:51:55 PM-> Applied to database : 1782171639
6/12/2020 7:51:55 PM-> Resolved locally : 2094907535
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20180103.2
6/12/2020 7:51:55 PM-> Applied to database : 1038704292
6/12/2020 7:51:55 PM-> Resolved locally : 1626189996
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20180103.5
6/12/2020 7:51:55 PM-> Applied to database : -757105328
6/12/2020 7:51:55 PM-> Resolved locally : 1831763030
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20180105.6
6/12/2020 7:51:55 PM-> Applied to database : -1745480510
6/12/2020 7:51:55 PM-> Resolved locally : -1638382870
6/12/2020 7:51:55 PMMigration checksum mismatch for migration version 2.20180105.10
6/12/2020 7:51:55 PM-> Applied to database : 1040883703
6/12/2020 7:51:55 PM-> Resolved locally : -1249201146

@Sourav1407
Copy link
Author

@Sourav1407 Sourav1407 commented Jun 12, 2020

@juliahayward Got this for ~100+ files. Was not getting this for 4.2.0. Even if I repair then I don't know the impact and how this encoding is going to impact the future migrations. If I repair the checksum the checksum value will change and if I see any issues with future migrations I will not be able to even rollback to 4.2.0

@juliahayward
Copy link
Member

@juliahayward juliahayward commented Jun 12, 2020

OK, then remove that encoding setting... we will need to expand the encoding functionality so that you can override it on a per-script basis rather than all or nothing. I'll try to implement that in the next patch version, which should be out next week.

@Sourav1407
Copy link
Author

@Sourav1407 Sourav1407 commented Jun 15, 2020

@juliahayward Sure Thanks

@Sourav1407
Copy link
Author

@Sourav1407 Sourav1407 commented Jun 27, 2020

@juliahayward @Lyeeedar Closing comments please. If I upgrade to latest version, will the issue be resolved? Any encoding I have to provide in config?

@juliahayward
Copy link
Member

@juliahayward juliahayward commented Jul 1, 2020

Yes. You will need to add a file DB_sql_V3_20190813_27__Portuguese_Generic_Utterances.sql.conf next to the relevant migration that contains simply:

encoding=ISO_8859_1

and then all should be good. You don't need to alter any previous scripts and the encoding change is not required in the main configuration.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants