Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add FITS header check as a user option #56

Open
JulienPeloton opened this issue Oct 19, 2018 · 0 comments
Open

Add FITS header check as a user option #56

JulienPeloton opened this issue Oct 19, 2018 · 0 comments
Assignees
Labels
enhancement New feature or request

Comments

@JulienPeloton
Copy link
Member

The PR #55 fixed a bug with the header check (the connector was checking all FITS header before starting the job. Good idea until you have +10,000 files....). The fix is however temporary, as you can see below:

// Check that all the files have the same Schema
// in order to perform the union. Return the HDU type.
// NOTE: This operation is very long for hundreds of files!
// NOTE: Limit that to the first 10 files.
// NOTE: Need to be fixed!
val implemented = if (listOfFitsFiles.size < 10) {
  checkSchemaAndReturnType(listOfFitsFiles)
} else{
  checkSchemaAndReturnType(listOfFitsFiles.slice(0, 10))
}

The idea would be then to add an option to the DataFrameReader (e.g. checkHeader), false by default, that would trigger the header checker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant