Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Handle LOBs that exist outside the database #346

Closed
2 tasks done
chalkos opened this issue Jan 10, 2019 · 2 comments
Closed
2 tasks done

Handle LOBs that exist outside the database #346

chalkos opened this issue Jan 10, 2019 · 2 comments
Assignees
@tomasmferreira
Labels
feature Issue type
Projects
Supplements to DBPTK by NAE
Milestone
v2.3.0

Comments

@chalkos
Copy link
Contributor

chalkos commented Jan 10, 2019 •
edited by tomasmferreira

If binary files are stored outside from the database (in file system) they should be handled like internal LOBs

  • base system to support the different types of external lobs
  • implementation: files on local filesystem
@chalkos chalkos added the feature Issue type label Jan 10, 2019
@chalkos chalkos added this to the v2.3.0 milestone Jan 10, 2019
@chalkos chalkos self-assigned this Jan 10, 2019
@chalkos chalkos added this to Current milestone in Supplements to DBPTK by NAE Apr 2, 2019
@chalkos chalkos assigned tomasmferreira and unassigned chalkos Apr 2, 2019
@chalkos chalkos moved this from Current milestone to In progress in Supplements to DBPTK by NAE Apr 3, 2019
@luis100
Copy link
Member

luis100 commented Apr 10, 2019

This will be implemented as a filter module. An abstract class will ease the development of new filter modules for different file access protocols (e.g. S3). A default implementation for file system access protocol will be implemented.

tomasmferreira added a commit that referenced this issue Apr 11, 2019
@tomasmferreira
Progress on #346 by adding base system to support the different types…
f9e1659 
… of external lobs
tomasmferreira added a commit that referenced this issue Apr 11, 2019
@tomasmferreira
Progress on #346 by implementing solution for files on local filesystem
fe3e4e0
tomasmferreira added a commit that referenced this issue Apr 12, 2019
@tomasmferreira
Fix #346 by adding system to support the different types of external …
cb7d15c 
…lobs

Implemented the option to add filter modules in between import and export modules when running dbptk:
   DBPTK help now shows available filter module names;
   Using -help <filter name> displays the list of parameters received by that filter;
   Using --filter or -f with the name of one or more filters (separated by ',') injects these filters (in the same order they are written in) into the workflow;
   Passing parameters to the filters is done using --filter<n>-<param long name> or -f<n><param short name> (where 'n' is the position of the filter in the list [starting at 1]).

Implemented base system to support different types of external lobs:
   Added filter module to support handling of external lobs which can be added to the workflow using '--filter external-lobs', this filter requires at least two parameters:
      '--filter<n>-column-list [path to file]' which points to the file containing the list of columns in the database that reference external lobs (this file follows the same syntax as the one exported from the 'list-tables' module;
      '--filter<n>-reference-type [reference type]' which defines the strategy used to fetch the external lobs (ex.: file-system);
   Implemented abstract class ExternalLOBSCellHandler to ease the development of solutions for different types of external lobs by simply initializing a implementation of this class and passing it to the ExternalLOBSFilter.

Implemented solution for external lobs on local file system
tomasmferreira added a commit that referenced this issue Apr 12, 2019
@tomasmferreira
Fix #346 by adding system to support the different types of external …
713b383 
…lobs

Implemented the option to add filter modules in between import and export modules when running dbptk:
   DBPTK help now shows available filter module names;
   Using -help <filter name> displays the list of parameters received by that filter;
   Using --filter or -f with the name of one or more filters (separated by ',') injects these filters (in the same order they are written in) into the workflow;
   Passing parameters to the filters is done using --filter<n>-<param long name> or -f<n><param short name> (where 'n' is the position of the filter in the list [starting at 1]).

Implemented base system to support different types of external lobs:
   Added filter module to support handling of external lobs which can be added to the workflow using '--filter external-lobs', this filter requires at least two parameters:
      '--filter<n>-column-list [path to file]' which points to the file containing the list of columns in the database that reference external lobs (this file follows the same syntax as the one exported from the 'list-tables' module;
      '--filter<n>-reference-type [reference type]' which defines the strategy used to fetch the external lobs (ex.: file-system);
   Implemented abstract class ExternalLOBSCellHandler to ease the development of solutions for different types of external lobs by simply initializing a implementation of this class and passing it to the ExternalLOBSFilter.

Implemented solution for external lobs on local file system
@tomasmferreira tomasmferreira moved this from In progress to Done in Supplements to DBPTK by NAE Apr 17, 2019
@luis100 luis100 mentioned this issue Apr 24, 2019
Closed
@Laurira
Copy link

Laurira commented May 3, 2019

Made first test with sakila database (mysql on windows)

I got java.lang.NullPointerException: null
	at sun.nio.fs.WindowsPathParser.parse(Unknown Source)
	at sun.nio.fs.WindowsPathParser.parse(Unknown Source)
	at sun.nio.fs.WindowsPath.parse(Unknown Source)
	at sun.nio.fs.WindowsFileSystem.getPath(Unknown Source)
	at sun.nio.fs.AbstractPath.resolve(Unknown Source)

This is how my database table with external lobs looks like
error

file path is C:\Users\laurir\Desktop\DBPTK\SakilaPictureFolder\a13c4e637be4f4a33331ebbfe9c7956c.jpg

Command that I executed

java "-Dfile.encoding=UTF-8" -jar .\dbptk-app-2.3.0-RC.jar -i mysql --import-hostname=localhost -idb sakila -iu root -ip parool -e siard-2 -ef sakila.siard -f external-lobs --filter1-reference-t
ype file-system --filter1-column-list .\sakilaListTablesWithExternalLobs

column list has one row in it
sakila.customer{avatarFilePath;}

Did I do something wrong?
At the moment the creation of siard fails when starting to convert the table customer.

@luis100 luis100 mentioned this issue May 3, 2019
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tomasmferreira tomasmferreira

Labels
feature Issue type
Projects
No open projects
Supplements to DBPTK by NAE
  
Done
Milestone
v2.3.0
Development

No branches or pull requests
4 participants
@chalkos @luis100 @Laurira @tomasmferreira