New yeoda filename #21

fl0roth · 2020-10-07T14:00:49Z

Implementation of the new filename and folder logic with a more compact and general purpose use. The corresponding changes include:

Introduction of a yeoda_naming class which represents the new filenaming and folder logic and a corresponding test_yeoda_naming routine.
Introduction of fields with variable length (defined by length = 0) in the SmartFilename and SmartFilenamePart class.
Introduction of an option (input parameter "compact") to use a compact filename design. If it is set to True, empty fields are represented by a single pad character and all fields are treated with variable length.
Adaption of the attached conda environment by updating the python-dateuntil modul (issue#18).

…nd set the date format within this methods. The relative orbit is written into the extra_field which accepts more general inputs. Adapted the decoding and encoding methods according to this new field definition.

…he new Yeoda filenaming.

…een date and datetime using the isinstance function in the right order. Adapted the encoding method of the datetime field by checking for the 'T' to get time information. Removed deconding and encoding methods for time.

…ion a length of zero.

… statement to catch the corresponding inputs.

…s a variable length field at the end of it. Implemented rule which requires a delimiter in case of a variable length field.

… length to the __init__ function of the SmartFilenamePart.

…s to match Yeoda naming class.

… errors in case of an empty field.

…ming scheme.

…ture.

…er separately.

…into new_YeodaFilename

…hfinder into new_YeodaFilename

…in case of smaller arguments compared to the defined length. Additionally, all empty fields are represented by a single pad. Adapted SmartFilenamePart class to handle this functionality.

…d if the argument is smaller then the expected length.

…compact option is True. Added if statement in the __set_item__ method to get this functionality.

…ing.

…hfinder into new_YeodaFilename

…limiter within a field with variable length.

…limiter within a field with variable length when reading filename.

bbauerma

Looks fine for me.
a) yeoda-filename works as expected
b) all tests on my local Windows machine are green

FelixReuss

works fine with S1 Sigma

cnavacch

Looks very good! Three changes still need to be done:

Fixing Python version (@bbauerma @fl0roth ?)
Change loop in _build_fn() (I will do this).
Add compact=False in the __init__ of all other file naming conventions. This might me a nice feature to have for the existing file naming conventions (@fl0roth ).

cnavacch · 2020-10-21T10:04:56Z

conda_env.yml

@@ -2,9 +2,9 @@ name: geopathfinder
 channels:
  - defaults
 dependencies:
-  - python=3.6
+  - python


I would prefer to restrict the Python version to e.g. python=3.7. Simply using python is too general.

actually, can we remove python at all?

it should work for python 3. for all python 3 versions (3.6, 3.7,..)

I would recommend >= 3.6

The issue I see at the moment is that we use the same conda yml file for Travis. I prefer to have separate yml files for testing each Python version with Travis. This ensures that each Python version works/is supported and is always tested and makes it easier to reset package dependencies for a specific Python version. In the main yml file I agree with @sebhahn , there we could set python >= 3.6.

is already set to python >= 3.6 now
please proceed and implement the steps you suggested - i want to release a new version...

src/geopathfinder/file_naming.py

…od of SmartFilenamePart to allow a more performant field concatenation;

… all existing file naming conventions.

cnavacch

Great, all changes are fine from my side!

bbauerma and others added 30 commits September 23, 2020 18:06

introduces YeodaFilename & co; relaxes rules on lenght of filename-parts

a40379f

Adapted setup function and test functions until test31 to work with t…

fed2ab1

…he new Yeoda filenaming.

Adapted start and end time functions to work with the datetime approach.

2510548

Added detection of fields with variable length which have per definit…

fa2078b

…ion a length of zero.

Adapted _build_fn method to handle empty zero length inputs. Added if…

049ca14

… statement to catch the corresponding inputs.

Solved problem which reads in a wrong length from filename if there i…

456b6ac

…s a variable length field at the end of it. Implemented rule which requires a delimiter in case of a variable length field.

Added rule to ensure there is a delimiter in case of a variable field…

808567b

… length to the __init__ function of the SmartFilenamePart.

Removed the date_delimiter field to make the filename easier readable.

754204c

Adapted test2 to test32 to Yeoda naming class. Adapted filename input…

b27c65e

…s to match Yeoda naming class.

Removed TODO string and interim results.

87eb763

Updated TODOs and doc strings.

bf58aff

Adapted Yeoda path function to match new folder structure.

4f1f558

Adapted Yeoda tree function to match new folder structure.

e2d1a10

Removed string function from the encoding of the extra_field to avoid…

c3c66df

… errors in case of an empty field.

Adapted test4 and 5 of the filenaming tests to match the new Yeoad na…

00c1cd0

…ming scheme.

Adapted tests for yeoda path and yeoda tree to match new folder struc…

bcacc58

…ture.

Solved problem of missing run number by checking version and run numb…

8819f8c

…er separately.

Adapted test data link to get the data from the Yeoda folder structure.

343f880

Closes #1 (logic in get_disk_usage was checked); Closes #8

1956de7

Merge branch 'new_YeodaFilename' of github.com:TUW-GEO/geopathfinder …

912d147

…into new_YeodaFilename

Altered python-dateutil version to match the required version of pandas.

910921c

Merge branch 'new_YeodaFilename' of https://github.com/TUW-GEO/geopat…

7b462f8

…hfinder into new_YeodaFilename

Introduced input option called compact which removes additional pads …

2d12d7d

…in case of smaller arguments compared to the defined length. Additionally, all empty fields are represented by a single pad. Adapted SmartFilenamePart class to handle this functionality.

Altered expected outcomes to match new compact filename approach.

9619ad4

Pass compact option to the parent class.

35db5f0

Improved compact option by ensure that no additional pads are include…

1bc4b02

…d if the argument is smaller then the expected length.

Added new test to check the compact filename approach.

c5741ed

Added folder structure of new Yeoda naming scheme to test data.

ebfb94b

fl0roth added 2 commits October 7, 2020 15:09

Ensure that the all fields are treated as zero length in case of the …

2b8c21d

…compact option is True. Added if statement in the __set_item__ method to get this functionality.

Changed all field lengths to zero.

b0ac637

fl0roth requested review from bbauerma, cnavacch and FelixReuss October 7, 2020 14:00

bbauerma and others added 9 commits October 15, 2020 15:40

repairs test path (is platform safe now)

1827c12

relaxes python-dateutil in yml (for travis test)

ff364ad

relaxes python in yml (for travis test)

44ba6bd

relaxes python in yml (for travis test) - attempt 2

d311613

relaxes python in yml (for travis test) - attempt 3. fixes #18

33fa811

Switched replacement of empty fields from single-Padding to empty str…

bc86612

…ing.

Merge branch 'new_YeodaFilename' of https://github.com/TUW-GEO/geopat…

34ff2f7

…hfinder into new_YeodaFilename

Added test to check the raise of an exception in case of a missing de…

c58db31

…limiter within a field with variable length.

Added test to check the raise of an exception in case of a missing de…

10f17e2

…limiter within a field with variable length when reading filename.

bbauerma approved these changes Oct 19, 2020

View reviewed changes

FelixReuss approved these changes Oct 21, 2020

View reviewed changes

cnavacch requested changes Oct 21, 2020

View reviewed changes

cnavacch and others added 3 commits October 21, 2020 15:51

moved compact check to __build_fn; modified __repr__ and __str__ meth…

a3a7bb3

…od of SmartFilenamePart to allow a more performant field concatenation;

declares python=3.6 in dependencies/yml

78b789d

Added compact=False option in __init__ and from_filename functions of…

2c2b592

… all existing file naming conventions.

fl0roth requested a review from cnavacch October 22, 2020 07:15

cnavacch added 6 commits October 23, 2020 17:36

updated travis to check conda env files for Python 3.6 and 3.7

0f1d52e

updated travis to check conda env files for Python 3.6 and 3.7

b3dffe6

updated travis yml file

4ed2f2a

defined stages in travis yml file

8a201ed

defining jobs in travis

93645e4

only report success to coveralls for the most recent version

dac4f77

cnavacch approved these changes Oct 23, 2020

View reviewed changes

bbauerma merged commit 6f4a86a into master Oct 28, 2020

bbauerma deleted the new_YeodaFilename branch October 28, 2020 17:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New yeoda filename #21

New yeoda filename #21

fl0roth commented Oct 7, 2020

bbauerma left a comment •

edited

Loading

FelixReuss left a comment

cnavacch left a comment

cnavacch Oct 21, 2020

bbauerma Oct 21, 2020

bbauerma Oct 21, 2020

sebhahn Oct 21, 2020

cnavacch Oct 22, 2020

bbauerma Oct 22, 2020

cnavacch left a comment

New yeoda filename #21

New yeoda filename #21

Conversation

fl0roth commented Oct 7, 2020

bbauerma left a comment • edited Loading

Choose a reason for hiding this comment

FelixReuss left a comment

Choose a reason for hiding this comment

cnavacch left a comment

Choose a reason for hiding this comment

cnavacch Oct 21, 2020

Choose a reason for hiding this comment

bbauerma Oct 21, 2020

Choose a reason for hiding this comment

bbauerma Oct 21, 2020

Choose a reason for hiding this comment

sebhahn Oct 21, 2020

Choose a reason for hiding this comment

cnavacch Oct 22, 2020

Choose a reason for hiding this comment

bbauerma Oct 22, 2020

Choose a reason for hiding this comment

cnavacch left a comment

Choose a reason for hiding this comment

bbauerma left a comment •

edited

Loading