ENH: Improve file parsing

[rstoneback]

Description

Addresses #763, #762

• Added support for user supplied variables in file template strings, not just those used for dates.
• Improved support for delimited filenames. More than one variable can be within a single delimited section. Delimited sections are not required to have a variable.
• Updated docstring in from_os as well as supporting methods.
• Added tests for delimited filenames, user variables, and wildcard use.
• Condensed existing tests
• Output from process_parsed_filenames is now always sorted

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide
instructions so we can reproduce. Please also list any relevant details for
your test configuration

• Verified parsing delimited filenames now works on COSMIC. COSMIC filenames violate a strict interpretation of delimited information. Two examples using the fixed_width parser, two examples using the delimited parser.

pysat.Files.from_os(data_path, '{year2:04d}.{day2:03d}/ionPrf_C{sat_id:03d}.{year:04d}.{day:03d}.{hour:02d}.{minute:02d}.???_????.????_nc')
pysat.Files.from_os(data_path, '{year2:04d}.{day2:03d}/ionPrf_C{sat_id:03d}.{year:04d}.{day:03d}.{hour:02d}.{minute:02d}.G{code:02d}_{year3:04d}.{radio:04d}_nc')
pysat.Files.from_os(data_path, '{year2:04d}.{day2:03d}/ionPrf_C{sat_id:03d}.{year:04d}.{day:03d}.{hour:02d}.{minute:02d}.{code:8}.{radio:04d}_nc', delimiter='.')
pysat.Files.from_os(data_path, '{year2:04d}.{day2:03d}/ionPrf_C{sat_id:03d}.{year:04d}.{day:03d}.{hour:02d}.{minute:02d}.*.{radio:04d}_nc', delimiter='.')

• Added unit tests

Test Configuration:

• Operating system: MacOS
• Version number: Python 3.9.7

Checklist:

• Make sure you are merging into the develop (not main) branch
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• Any dependent changes have been merged and published in downstream modules
• Add a note to CHANGELOG.md, summarizing the changes

If this is a release PR, replace the first item of the above checklist with the release
checklist on the wiki: https://github.com/pysat/pysat/wiki/Checklist-for-Release

[rstoneback]

Tests are failing due to some Meta thing which I believe @jklenzing has already identified. The pull request forms are passing.

[aburrell]

The figure you use in the example doesn't show the entire command you use for testing. Can you include that as a code snippet?

[aburrell]

Haven't tried things out yet, but looked over the code.

[aburrell]

Ok, can confirm this does NOT work if you use the * in the middle of the specifiers. You can use a * on the front end, though.

[aburrell]

Suggested change

	# engage the fixed width filename parser. If there is not a common
	# delimiter then the fixed width parser is suggested though not always
	# required. Given the range of standards compliance across the decades of
	# engage the fixed width filename parser. Given the range of standards
	# compliance across the decades of

And fix spacing. Recommend removing that sentence because from this level, you can't specify which parser is used. I was reading this today and it took me a moment before I realized I had no control over that aspect.

[rstoneback]

A developer can control which parser is used when starting here as a template. If delimiter=None or left unassigned then the fixed width parser is used.

I don't understand the fix spacing comment.

[aburrell]

"If there is not a common delimiter then the fixed width parser is suggested though not always required" is not accurate in this setting because you can ONLY differentiate between the two by providing or not providing a delimiter.

"fix spacing" means that I don't know if the lines are now too long or short as a part of the whole comment block.

[rstoneback]

It is correct that you can only provide a single given delimiter or use the fixed width parser. In practice though, the delimited parser will likely work on filenames that don't have even a single instance of the given delimiter. I think I even test for that.

"fix spacing" means that I don't know if the lines are now too long or short as a part of the whole comment block.

Got it!

[aburrell]

GitHub shows me that this hasn't been addressed? Which is confusing because this is marked as outdated.

[rstoneback]

Sometimes I think it gets confused. I updated the comment a bit to specifically note I am talking about modifying the function call from_os below the comment.

[rstoneback]

Ok, can confirm this does NOT work if you use the * in the middle of the specifiers. You can use a * on the front end, though.

You can use the '?' in the middle of specifiers though. Requires knowing ahead of time how many characters are needed.

[aburrell]

Requires knowing ahead of time how many characters are needed.

Also requires that the number of characters is always the same. For both reasons, this is why I couldn't use it.