Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Disable recursive filesystem scanning down hidden directories #1807

Closed
nelvadas opened this issue Oct 25, 2017 · 3 comments
Closed

Disable recursive filesystem scanning down hidden directories #1807

nelvadas opened this issue Oct 25, 2017 · 3 comments

Comments

@nelvadas
Copy link

@nelvadas nelvadas commented Oct 25, 2017

What version of Flyway are you using?

4.2.0
Which client are you using? (Command-line, Java API, Maven plugin, Gradle plugin, SBT plugin, ANT tasks)

Command line
What database are you using (type & version)?

Postgres 9.6
What operating system are you using?

Containers:
Migration container Alpine Linux
DB container: centos
What did you do?

(Please include the content causing the issue, any relevant configuration settings, and the command you ran)
Trying to migrate apply a simple SQL Migration on a postgres DB from openshift, The data SQL are provided by a mounted volume .Files seems replicated at runtime and the migration failed with the error

In my /var/flyway/data I provided only one file.

When connecting to the migraiton container and run the command manually in debug mode i have htis

$/opt/flyway/4.2.0 # ./flyway info -user=root -password=Europa01# -url=jdbc:postgresql://beosbank-posgres-db-europa/beosbank-europa -locations=filesystem:/var/
flyway/data -X
Flyway 4.2.0 by Boxfuse

DEBUG: Java 1.8.0_131 (Oracle Corporation)
DEBUG: Linux 3.10.0-514.6.1.el7.x86_64 amd64

DEBUG: Loading config file: /opt/flyway/4.2.0/conf/flyway.conf
DEBUG: Unable to load config file: /root/flyway.conf
DEBUG: Unable to load config file: /opt/flyway/4.2.0/flyway.conf
DEBUG: Using configuration:
DEBUG: flyway.jarDirs -> /opt/flyway/4.2.0/jars
DEBUG: flyway.password -> *********
DEBUG: flyway.url -> jdbc:postgresql://beosbank-posgres-db-europa/beosbank-europa
DEBUG: flyway.user -> root
DEBUG: flyway.locations -> filesystem:/var/flyway/data
DEBUG: Adding location to classpath: /opt/flyway/4.2.0/drivers/derby-10.12.1.1.jar
DEBUG: Adding location to classpath: /opt/flyway/4.2.0/drivers/derbyclient-10.12.1.1.jar
DEBUG: Adding location to classpath: /opt/flyway/4.2.0/drivers/greenplumdriver-5.1.1.jar
DEBUG: Adding location to classpath: /opt/flyway/4.2.0/drivers/h2-1.4.193.jar
DEBUG: Adding location to classpath: /opt/flyway/4.2.0/drivers/hsqldb-2.3.4.jar
DEBUG: Adding location to classpath: /opt/flyway/4.2.0/drivers/jtds-1.3.1.jar
DEBUG: Adding location to classpath: /opt/flyway/4.2.0/drivers/mariadb-java-client-1.4.5.jar
DEBUG: Adding location to classpath: /opt/flyway/4.2.0/drivers/postgresql-9.4.1208.jre6.jar
DEBUG: Adding location to classpath: /opt/flyway/4.2.0/drivers/sqlite-jdbc-3.7.15-M1.jar
Database: jdbc:postgresql://beosbank-posgres-db-europa/beosbank-europa (PostgreSQL 9.6)
DEBUG: DDL Transactions Supported: true
DEBUG: Schema: public
DEBUG: Spring Jdbc available: false
DEBUG: Scanning for SQL callbacks ...
DEBUG: Scanning for filesystem resources at '/var/flyway/data' (Prefix: '', Suffix: '.sql')
DEBUG: Scanning for resources in path: /var/flyway/data (/var/flyway/data)
DEBUG: Scanning for resources in path: /var/flyway/data/..data (/var/flyway/data)
DEBUG: Scanning for resources in path: /var/flyway/data/..109810_21_10_22_56_31.560272816 (/var/flyway/data)
DEBUG: Found filesystem resource: /var/flyway/data/..109810_21_10_22_56_31.560272816/V3.0__UpdateStreet.sql
DEBUG: Found filesystem resource: /var/flyway/data/..data/V3.0__UpdateStreet.sql
DEBUG: Found filesystem resource: /var/flyway/data/V3.0__UpdateStreet.sql
DEBUG: Update Count: 0
DEBUG: Scanning for filesystem resources at '/var/flyway/data' (Prefix: 'V', Suffix: '.sql')
DEBUG: Scanning for resources in path: /var/flyway/data (/var/flyway/data)
DEBUG: Scanning for resources in path: /var/flyway/data/..data (/var/flyway/data)
DEBUG: Scanning for resources in path: /var/flyway/data/..109810_21_10_22_56_31.560272816 (/var/flyway/data)
DEBUG: Found filesystem resource: /var/flyway/data/..109810_21_10_22_56_31.560272816/V3.0__UpdateStreet.sql
DEBUG: Found filesystem resource: /var/flyway/data/..data/V3.0__UpdateStreet.sql
DEBUG: Found filesystem resource: /var/flyway/data/V3.0__UpdateStreet.sql
DEBUG: Scanning for filesystem resources at '/var/flyway/data' (Prefix: 'R', Suffix: '.sql')
DEBUG: Scanning for resources in path: /var/flyway/data (/var/flyway/data)
DEBUG: Scanning for resources in path: /var/flyway/data/..data (/var/flyway/data)
DEBUG: Scanning for resources in path: /var/flyway/data/..109810_21_10_22_56_31.560272816 (/var/flyway/data)
DEBUG: Filtering out resource: /var/flyway/data/..109810_21_10_22_56_31.560272816/V3.0__UpdateStreet.sql (filename: V3.0__UpdateStreet.sql)
DEBUG: Filtering out resource: /var/flyway/data/..data/V3.0__UpdateStreet.sql (filename: V3.0__UpdateStreet.sql)
DEBUG: Filtering out resource: /var/flyway/data/V3.0__UpdateStreet.sql (filename: V3.0__UpdateStreet.sql)
ERROR: Unexpected error
org.flywaydb.core.api.FlywayException: Found more than one migration with version 3.0
Offenders:
-> /var/flyway/data/..109810_21_10_22_56_31.560272816/V3.0__UpdateStreet.sql (SQL)
-> /var/flyway/data/V3.0__UpdateStreet.sql (SQL)
at org.flywaydb.core.internal.resolver.CompositeMigrationResolver.checkForIncompatibilities(CompositeMigrationResolver.java:142)
at org.flywaydb.core.internal.resolver.CompositeMigrationResolver.doFindAvailableMigrations(CompositeMigrationResolver.java:108)
at org.flywaydb.core.internal.resolver.CompositeMigrationResolver.resolveMigrations(CompositeMigrationResolver.java:91)
at org.flywaydb.core.internal.resolver.CompositeMigrationResolver.resolveMigrations(CompositeMigrationResolver.java:44)
at org.flywaydb.core.internal.info.MigrationInfoServiceImpl.refresh(MigrationInfoServiceImpl.java:112)
at org.flywaydb.core.Flyway$4.execute(Flyway.java:1115)
at org.flywaydb.core.Flyway$4.execute(Flyway.java:1097)
at org.flywaydb.core.Flyway.execute(Flyway.java:1464)
at org.flywaydb.core.Flyway.info(Flyway.java:1097)
at org.flywaydb.commandline.Main.executeOperation(Main.java:152)
at org.flywaydb.commandline.Main.main(Main.java:103)
/opt/flyway/4.2.0 #
What did you expect to see?

Migration successful
What did you see instead?

Unexpected file in the migration process
DEBUG: Found filesystem resource: /var/flyway/data/..109810_21_10_22_56_31.560272816/V3.0__UpdateStreet.sql

In my folder i put only one file V3.0__UpdateStreet.sql

RROR: Unexpected error
org.flywaydb.core.api.FlywayException: Found more than one migration with version 3.0
Offenders:
-> /var/flyway/data/..109810_21_10_22_56_31.560272816/V3.0__UpdateStreet.sql (SQL)
-> /var/flyway/data/V3.0__UpdateStreet.sql (SQL)
at org.flywaydb.core.internal.resolver.CompositeMigrationResolver.checkForIncompatibilities(CompositeMigrationResolver.java:142)
at org.flywaydb.core.internal.resolver.CompositeMigrationResolver.doFindAvailableMigrations(CompositeMigrationResolver.java:108)
at org.flywaydb.core.internal.resolver.CompositeMigrationResolver.resolveMigrations(CompositeMigrationResolver.java:91)
at org.flywaydb.core.internal.resolver.CompositeMigrationResolver.resolveMigrations(CompositeMigrationResolver.java:44)
at org.flywaydb.core.internal.info.MigrationInfoServiceImpl.refresh(MigrationInfoServiceImpl.java:112)
at org.flywaydb.core.Flyway$4.execute(Flyway.java:1115)
at org.flywaydb.core.Flyway$4.execute(Flyway.java:1097)
at org.flywaydb.core.Flyway.execute(Flyway.java:1464)
at org.flywaydb.core.Flyway.info(Flyway.java:1097)
at org.flywaydb.commandline.Main.executeOperation(Main.java:152)
at org.flywaydb.commandline.Main.main(Main.java:103)
/opt/flyway/4.2.0

#1801

While browsing the data directory i can see only one file .

/var/flyway/data # ls
V1_1__updateAddress.sql
/var/flyway/data # ls -a
. ..109810_25_10_13_19_58.236419878 V1_1__updateAddress.sql
.. ..data
/var/flyway/data # ls
V1_1__updateAddress.sql
/var/flyway/data #

nelvadas pushed a commit to nelvadas/ocp-flyway-db-migration that referenced this issue Oct 26, 2017
@axelfontaine
Copy link
Contributor

@axelfontaine axelfontaine commented Oct 26, 2017

This is an issue with your environment. Flyway works as designed.

@tsloughter
Copy link

@tsloughter tsloughter commented Nov 16, 2017

I just encountered this as well. The design of flyway is to consider hidden directories in recursive search? It wasn't mentioned in the documentation for locations and there doesn't appear to be a way to at least tell it not to recurse directories.

@axelfontaine axelfontaine changed the title hidden files displayed while listing sql files recursively => duplicate migrations Add a flag to disable recursive filesystem/classpath scanning Nov 16, 2017
@axelfontaine axelfontaine added this to the Flyway 5.0.0 milestone Nov 16, 2017
@axelfontaine axelfontaine reopened this Nov 16, 2017
@olib963
Copy link

@olib963 olib963 commented Aug 26, 2018

I too have encountered this issue. I am trying to run flyway as an init container in my kubernetes deployment by simply using the official docker image and mounting my SQL files from a configmap onto that image.

The problem is that when kubernetes mounts the configmap containing the SQL migrations it mounts hidden files used by k8s to keep the config up to date. So if you had a directory containing only one file V1__create_initial_tables.sql then what will be mounted currently is:

/flyway/sql/V1__create_initial_tables.sql
/flyway/sql/..data/V1__create_initial_tables.sql
/flyway/sql/..$(TIMESTAMP).$(ID)/V1__create_initial_tables.sql

with the last hidden directories being linked.

It would seem that if you specify flyway to search /flyway/sql/ it will recursively find hidden directories however if you Specify the hidden directory as a location /flyway/sql/..data/ then flyway cannot resolve that directory.

I have come up with a work around of creating a custom docker image for each micro service that simply copies the SQL directory onto the image, but it would be nice to simply have the pattern of mounting the SQL files onto the official image.

Perhaps adding an option to not recurse over hidden directories would be a sufficient solution? Please let me know if this would be acceptable :D

@axelfontaine axelfontaine changed the title Add a flag to disable recursive filesystem/classpath scanning Disable recursive filesystem scanning down hidden directories Sep 25, 2018
axelfontaine added a commit to flyway/flywaydb.org that referenced this issue Sep 25, 2018
axelfontaine added a commit that referenced this issue Sep 25, 2018
dohrayme pushed a commit to dohrayme/flyway that referenced this issue Feb 3, 2020
dohrayme pushed a commit to dohrayme/flyway that referenced this issue Feb 3, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants
You can’t perform that action at this time.