Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DM-1579: HTM indexed files. #42

Merged
merged 8 commits into from
Apr 1, 2016
Merged

DM-1579: HTM indexed files. #42

merged 8 commits into from
Apr 1, 2016

Commits on Mar 31, 2016

  1. Implement simple spacial indexer

    This implements an indexer that provides the methods necessary to
    shard a catalog and then look up the correct shard later.  The package
    I use currently has a problem if you call the lookup_id method without
    first calling another function.  This hack will go away if the package
    is fixed or if we get an LSST maintained version.
    SimonKrughoff committed Mar 31, 2016
    Configuration menu
    Copy the full SHA
    b36d318 View commit details
    Browse the repository at this point in the history
  2. Add a task for reading text files

    This will be used to read files from disk for indexing
    SimonKrughoff committed Mar 31, 2016
    Configuration menu
    Copy the full SHA
    9d1d1d8 View commit details
    Browse the repository at this point in the history
  3. Add config for the catalog ingester

    The catalog ingester will load files using a helper task
    and put the contents in a set of SourceCatalogs, one
    per HTM trixel.  Most of the config is to allow mapping of
    columns in the file to the schema needed for referenc catalogs.
    SimonKrughoff committed Mar 31, 2016
    Configuration menu
    Copy the full SHA
    0d2cb01 View commit details
    Browse the repository at this point in the history
  4. Implement the reference catalog ingester

    This implements the task to take files and parse them into
    SourceCatalogs.  Some of the tasks implemented here are:
    read the file into a numpy array (the whole file needs to fit
    in memory), construct the SourceCatalog schema from the array
    dtype, index the catalog and fill the SourceCatalog.
    
    Command line tasks need a command runner.  Since this task
    doesn't need to parse an `--id` argument, we an forego that
    step.  Essentially, this just calls the task run method and
    gives a place to persist the config.
    SimonKrughoff committed Mar 31, 2016
    Configuration menu
    Copy the full SHA
    57d0dc1 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    2d5acdf View commit details
    Browse the repository at this point in the history

Commits on Apr 1, 2016

  1. Implement the catalog loader

    This involves reconstituting the task that ingested the files.
    
    The only config is the name of the config to read to create the
    ingester task.  The loader then borrows the indexer from the
    ingester task to grab the files from the repo it needs to load the
    reference catalog.  The shards are only clipped if they land on the
    boundary of the circular aperture.
    
    This is a minimal override of the reference catalog loader.  It may
    be possible to optimize by overriding other default implementations.
    SimonKrughoff committed Apr 1, 2016
    Configuration menu
    Copy the full SHA
    5f09f1c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    c21ed6d View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    c56e30d View commit details
    Browse the repository at this point in the history