Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add dispatcher utility #37

Merged
merged 18 commits into from Sep 25, 2019

Conversation

@mraspaud
Copy link
Member

commented Sep 11, 2019

Add the dispatcher utility.

client1:
  host: ftp://ftp.client1.com
  connection_parameters:
    credential_file: .netrc
  filepattern: '{platform_name}_{start_time}.{format}'
  directory: /input_data/{sensor}
  dispatch:
    - topics:
        - /level2/viirs
        - /level2/avhrr
      conditions:
        # key matches metadata items or provides default
        - product: [green_snow, true_color]
          sensor: viirs
        - product: [green_snow, overview]
          sensor: avhrr
          # special section "except" for negating
          except:
            platform_name: NOAA-15
    - topics:
        - /level3/cloudtype
      directory: /input/cloud_products
      conditions:
        - area: omerc_bb
          daylight: '<30'
          coverage: '>50'
@mraspaud mraspaud requested a review from pnuu Sep 11, 2019
@mraspaud mraspaud self-assigned this Sep 11, 2019
mraspaud added 6 commits Sep 11, 2019
@codecov

This comment has been minimized.

Copy link

commented Sep 12, 2019

Codecov Report

❗️ No coverage uploaded for pull request base (master@358a759). Click here to learn what that means.
The diff coverage is 65.19%.

Impacted file tree graph

@@            Coverage Diff            @@
##             master      #37   +/-   ##
=========================================
  Coverage          ?   29.82%           
=========================================
  Files             ?       12           
  Lines             ?     1750           
  Branches          ?        0           
=========================================
  Hits              ?      522           
  Misses            ?     1228           
  Partials          ?        0
Impacted Files Coverage Δ
trollmoves/server.py 0% <0%> (ø)
setup.py 0% <0%> (ø)
trollmoves/utils.py 64.47% <100%> (ø)
trollmoves/movers.py 28.04% <28.04%> (ø)
trollmoves/hooks.py 75% <75%> (ø)
trollmoves/dispatcher.py 89.67% <89.67%> (ø)
trollmoves/tests/test_dispatcher.py 98.06% <98.06%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 358a759...168697a. Read the comment docs.

@pnuu
pnuu approved these changes Sep 12, 2019
Copy link
Contributor

left a comment

LGTM. Couple of typos in the docstrings

trollmoves/dispatcher.py Outdated Show resolved Hide resolved
trollmoves/dispatcher.py Outdated Show resolved Hide resolved
@pnuu

This comment has been minimized.

Copy link
Contributor

commented Sep 12, 2019

Oh, and inotify needs to be added to install_requires.

setup.py Show resolved Hide resolved
@mraspaud mraspaud marked this pull request as ready for review Sep 12, 2019
mraspaud added 8 commits Sep 12, 2019
@pnuu

This comment has been minimized.

Copy link
Contributor

commented Sep 23, 2019

I'll give some more comments tomorrow morning.

trollmoves/dispatcher.py Outdated Show resolved Hide resolved
trollmoves/dispatcher.py Outdated Show resolved Hide resolved
@pnuu

This comment has been minimized.

Copy link
Contributor

commented Sep 24, 2019

Could there be a condition for filename patterns? I'm thinking about messages coming from trollmoves transfers, which might need additional transfers to servers without trollmoves client capabilities, like external customers. Although now that I look at those messages I think there might be enought information even without this feature 🤔

@mraspaud

This comment has been minimized.

Copy link
Member Author

commented Sep 24, 2019

I was hoping to be free from the filenames and just work with metadata...

@pnuu

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2019

Isn't filename (or uid/uri) also metadata? 😁 Anyway, this can also be handled by having specific topics.

@pnuu

This comment has been minimized.

Copy link
Contributor

commented Sep 25, 2019

LGTM, feel free to merge if you don't find anything to change.

@mraspaud

This comment has been minimized.

Copy link
Member Author

commented Sep 25, 2019

Thanks. I'll test my current dispatch chains with the modifications to make sure it still works

@mraspaud mraspaud merged commit 4d22fe7 into pytroll:master Sep 25, 2019
4 checks passed
4 checks passed
Travis CI - Pull Request Build Passed
Details
codecov/patch No report found to compare against
Details
codecov/project No report found to compare against
Details
stickler-ci No lint errors found
Details
@mraspaud mraspaud deleted the mraspaud:feature-dispatcher branch Sep 25, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.