Add an s3stalker daemon #114

adybbroe · 2022-08-17T12:39:46Z

Add a runner or demon to stalk for new files in an S3 bucket based on the s3stalker script

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

codecov · 2022-08-17T12:41:46Z

Codecov Report

Merging #114 (456c73b) into main (62a1d71) will increase coverage by 0.69%.
The diff coverage is 99.64%.

@@            Coverage Diff             @@
##             main     #114      +/-   ##
==========================================
+ Coverage   90.52%   91.22%   +0.69%     
==========================================
  Files          22       27       +5     
  Lines        3577     3998     +421     
==========================================
+ Hits         3238     3647     +409     
- Misses        339      351      +12

Flag	Coverage Δ
unittests	`91.22% <99.64%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
pytroll_collectors/tests/test_fsspec_to_message.py	`93.29% <ø> (ø)`
pytroll_collectors/tests/test_s3stalker.py	`99.46% <99.21%> (-0.54%)`	⬇️
pytroll_collectors/s3stalker.py	`100.00% <100.00%> (ø)`
pytroll_collectors/s3stalker_daemon_runner.py	`100.00% <100.00%> (ø)`
pytroll_collectors/scisys.py	`62.18% <100.00%> (ø)`
...l_collectors/tests/test_s3stalker_daemon_runner.py	`100.00% <100.00%> (ø)`

... and 4 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

pnuu

Some comments, questions and suggestions :-)

bin/s3stalker_daemon.py

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Co-authored-by: Panu Lahtinen <pnuu+git@iki.fi>

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

…llectors into add-s3stalker-daemon # Conflicts: # bin/s3stalker_daemon.py Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

adybbroe · 2022-12-20T13:09:12Z

Damn. Moving the self._setup_and_start_communication()from the init to the run method as suggested was a bad idea I am afraid, it gives me this error:
ValueError: signal only works in main thread of the main interpreter
So, I am moving it back into the init @pnuu

adybbroe · 2022-12-20T13:12:00Z

Damn. Moving the self._setup_and_start_communication()from the init to the run method as suggested was a bad idea I am afraid, it gives me this error: ValueError: signal only works in main thread of the main interpreter So, I am moving it back into the init @pnuu

Oh, I see now, that you only suggested moving the publisher creation to the run method, sorry will redo...

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

# Conflicts: # .github/workflows/ci.yaml

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Split s3stalker and daemon parts

pnuu

Some new comments.

bin/s3stalker_daemon.py

pnuu · 2023-03-03T06:58:05Z

pytroll_collectors/s3stalker.py

@@ -54,6 +149,7 @@ class DatetimeHolder:

 def get_last_files(path, *args, pattern=None, **kwargs):
    """Get the last files from path (s3 bucket and directory)."""
+    kwargs['skip_instance_cache'] = True


It just seemed a bit odd to modify the kwargs as the first thing in the function. Would it be more clear that it is set where the call to this function is so there are no side-effects when the call is made? I'm fine also with this, maybe a short explanation would be nice.

pytroll_collectors/s3stalker.py

pytroll_collectors/s3stalker_daemon_runner.py

setup.py

doc/source/index.rst

Co-authored-by: Panu Lahtinen <pnuu+git@iki.fi>

mraspaud · 2023-03-03T10:05:06Z

@pnuu I have addressed or answered your comments...

doc/source/index.rst

bin/s3stalker_daemon.py

Co-authored-by: Martin Raspaud <martin.raspaud@smhi.se>

pnuu · 2023-03-09T13:25:40Z

I went through the unresolved comments, and only my two latest ones are still open.

For some reason GitHub shows some of the comments without the ability to resolve them, so the comment section is a bit messy.

pnuu

LGTM, thanks for the updates!

Just one suggestion on the S3 credential configuration.

doc/source/index.rst

Co-authored-by: Panu Lahtinen <pnuu+git@iki.fi>

Add an s3stalker daemon

62575b8

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

adybbroe self-assigned this Aug 17, 2022

Adam.Dybbroe added 2 commits August 18, 2022 10:30

Rearrange, add more logging, and clear instance cache

c1b38d8

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Add examples config files needed to run the s3stalker daemon

7e72e23

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

adybbroe marked this pull request as ready for review August 18, 2022 08:32

adybbroe requested review from TAlonglong and mraspaud August 18, 2022 08:33

adybbroe added the enhancement label Aug 18, 2022

Make it possible to specify start-up time window on command line

99b17b9

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

pnuu requested changes Sep 2, 2022

View reviewed changes

This was referenced Sep 14, 2022

Add S3 configuration to documentation #119

Open

Check for existing segments on S3 storage #117

Merged

Adam.Dybbroe and others added 6 commits December 19, 2022 12:00

Refactoring, moving script functionality in under the package

00033ea

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Update bin/s3stalker_daemon.py

0b01bd8

Co-authored-by: Panu Lahtinen <pnuu+git@iki.fi>

simplify and make more compact but still readable

9fc9e17

Co-authored-by: Panu Lahtinen <pnuu+git@iki.fi>

Refactoring

c98c102

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Merge branch 'add-s3stalker-daemon' of github.com:adybbroe/pytroll-co…

df1c4a9

…llectors into add-s3stalker-daemon # Conflicts: # bin/s3stalker_daemon.py Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Simplify and condense code following reviewers suggestions

32dddeb

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Adam.Dybbroe added 9 commits December 20, 2022 15:51

Refactyor and increase test coverage

48113ae

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Adding missing item topic to example config file

0d9e069

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Fix test-requires package list

16279d2

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Try fix github ci environment

d0eee4a

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Try fix CI env on github

2e4c956

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Try fix github ci env

5c2a61f

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Merge branch 'main' into add-s3stalker-daemon

af82b00

# Conflicts: # .github/workflows/ci.yaml

Try fix ci

6de58fe

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

Add pyorbital to ci env

84d74be

Signed-off-by: Adam.Dybbroe <a000680@c21856.ad.smhi.se>

mraspaud and others added 10 commits February 14, 2023 18:18

Split s3stalker and daemon parts

1b91eb2

Refactor s3stalker tests

91d5a25

Refactor s3stalker tests

abd10e1

Remove some implementation tests

ae66213

Make startup time a configuration item

c18a8ab

Remove s3stalker daemon's own time recording

1e4a64f

Cleanup daemon runner

5ceb5e6

Cleanup daemon runner

685975c

Document s3stalkers

8da86da

Merge pull request #1 from mraspaud/feature-s3stalker-daemon

4e7fcfd

Split s3stalker and daemon parts

pnuu reviewed Mar 3, 2023

View reviewed changes

mraspaud reviewed Mar 3, 2023

View reviewed changes

doc/source/index.rst Outdated Show resolved Hide resolved

mraspaud and others added 9 commits March 3, 2023 08:57

Update doc/source/index.rst

c19168a

Update pytroll_collectors/s3stalker.py

c05e352

Co-authored-by: Panu Lahtinen <pnuu+git@iki.fi>

Fix required python version

859a0da

Remove hardcoded option

ff68304

Simplify poll waiting

47f9a6d

Rename variable to better name

8ed0509

Remove unneeded logging

2839db3

Clean up get_last_files

20b0c1c

Update bin/s3stalker_daemon.py

7f26e03

Co-authored-by: Panu Lahtinen <pnuu+git@iki.fi>

pnuu reviewed Mar 6, 2023

View reviewed changes

doc/source/index.rst Outdated Show resolved Hide resolved

bin/s3stalker_daemon.py Show resolved Hide resolved

Update examples/s3stalker_log.yaml_template

5a5926f

Co-authored-by: Martin Raspaud <martin.raspaud@smhi.se>

mraspaud added 2 commits March 20, 2023 12:33

Allow passing the bucket in the configuration file

e9b6f3e

Link to example configs for s3stalkers

a0fc7dc

pnuu approved these changes Mar 20, 2023

View reviewed changes

doc/source/index.rst Show resolved Hide resolved

Update doc/source/index.rst

456c73b

Co-authored-by: Panu Lahtinen <pnuu+git@iki.fi>

pnuu merged commit 5b40ded into pytroll:main Mar 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an s3stalker daemon #114

Add an s3stalker daemon #114

adybbroe commented Aug 17, 2022

codecov bot commented Aug 17, 2022 •

edited

Loading

pnuu left a comment

adybbroe commented Dec 20, 2022

adybbroe commented Dec 20, 2022

pnuu left a comment

pnuu Mar 3, 2023

mraspaud commented Mar 3, 2023

pnuu commented Mar 9, 2023

pnuu left a comment

Add an s3stalker daemon #114

Add an s3stalker daemon #114

Conversation

adybbroe commented Aug 17, 2022

codecov bot commented Aug 17, 2022 • edited Loading

Codecov Report

pnuu left a comment

Choose a reason for hiding this comment

adybbroe commented Dec 20, 2022

adybbroe commented Dec 20, 2022

pnuu left a comment

Choose a reason for hiding this comment

pnuu Mar 3, 2023

Choose a reason for hiding this comment

mraspaud commented Mar 3, 2023

pnuu commented Mar 9, 2023

pnuu left a comment

Choose a reason for hiding this comment

codecov bot commented Aug 17, 2022 •

edited

Loading