Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Validate missing collections in bundle after CCB-282 updates #310

Closed
jordanpadams opened this issue Mar 22, 2021 · 7 comments · Fixed by #320
Closed

Validate missing collections in bundle after CCB-282 updates #310

jordanpadams opened this issue Mar 22, 2021 · 7 comments · Fixed by #320
Assignees
@qchaupds
Labels
B12.0 bug Something isn't working s.medium
Milestone
06.Mary.Decker.Sl...

Comments

@jordanpadams
Copy link
Member

Describe the bug
See comment from @rchenatjpl at #192 (comment)
@jordanpadams jordanpadams added bug Something isn't working needs:triage labels Mar 22, 2021
@jordanpadams jordanpadams added this to the 05.Gail.Devers milestone Mar 22, 2021
@jordanpadams jordanpadams changed the title Issue parsing collections per CCB-282 updates Validate missing collections in bundle after CCB-282 updates Mar 24, 2021
@jordanpadams jordanpadams removed their assignment Mar 24, 2021
@qchaupds
Copy link
Contributor

I am wondering if the name patterns are too strict. @jordanpadams Can you clarify? The SDR mentions bundle but quiet on collection page 14.

vi ./src/main/java/gov/nasa/pds/tools/validate/BundleManager.java

public class BundleManager {
private static final Logger LOG = LoggerFactory.getLogger(BundleManager.class);
private static final Pattern COLLECTION_LABEL_PATTERN =
Pattern.compile("(.)*collection(.)\.xml", Pattern.CASE_INSENSITIVE);
private static final Pattern BUNDLE_LABEL_PATTERN =
Pattern.compile("(.)*bundle(.)\.xml", Pattern.CASE_INSENSITIVE);

From what I can gather, the code looks for files that has "bundle_" or "collection_" which would explain why when Richard changed the name, it doesn't complain because it was found in the crawler.

@qchaupds
Copy link
Contributor

On page 15, there is an example of a collection that has the name "collection_browse.xml"

@jordanpadams
Copy link
Member Author

@qchaupds we want to look for any file that is bundle*.xml for bundles and and collection*.xml for collections.

@tloubrieu-jpl
Copy link
Member

@qchaupds is looking at alternatives to crawl for the bundle and collection labels using apache library (fileUtil).

@qchaupds
Copy link
Contributor

The first listFiles() function here:

https://commons.apache.org/proper/commons-io/javadocs/api-2.5/org/apache/commons/io/FileUtils.html

with IOFilter as a parameter is causing the code to pause (possibly in an infinite loop). An alternate function listFiles() with extensions as an array of file extension ("xml") seems to work and get around the pause issue. Will give that a try to see the regression tests will run successful and the work for issue #310 can proceed.

qchaupds pushed a commit that referenced this issue Mar 30, 2021
Allows the name of a collection to not require an underscore immediat…
84da1cb 
…ely after the "collection" token.

1. Add github310 to src/test/resources directory.
2. Use alternate listFiles() function to avoid infinite loop in BundleManager.java
3. Ease requirement for an underscore immediately after the "collection" token in BundleManager.java
4. Commented out extraneous println statements in FieldValueValidator.java
5. Add function signature for crawl() to use list of file extensions in Crawler.java
6. Add function to craw() to use list of file extensions in FileCrawler.java
7. Consolidate a block of code to refinedFoundList() function so it can be reused in FileCrawler.java
8. Ease requirement for an underscore immediately after the "collection" token in CollectionReferentialIntegrityRule.java
9. Use alternate listFiles() function to avoid infinite loop in CollectionReferentialIntegrityRule.java
10. Ease requirement for an underscore immediately after the "collection" token in FindUnreferencedIdentifiers.java
11. Use alternate listFiles() function to avoid infinite loop in FindUnreferencedIdentifiers.java
12. Add 3 new constants to consolidate the locations of patterns to one place in Constants.java
13. Add two new tests for github310 in feature file validate.feature

Refs:

#310 Validate missing collections in bundle after CCB-282 updates
@qchaupds qchaupds mentioned this issue Mar 30, 2021
Merged
@jordanpadams jordanpadams added B12.0 and removed B11.1 labels Apr 1, 2021
qchaupds pushed a commit that referenced this issue Apr 1, 2021
PR Response 1: Make use of defined constants in one place to avoid hu…
12b7284 
…nting in many places

1.  Use defined constants in and remove un-needed comments in BundleManager.java
2.  Remove un-needed comments in FieldValueValidator.java
3.  Remove un-used codes in FileCrawler.java
4.  Use defined constants in CollectionReferentialIntegrityRule.java
5.  Remove un-needed comments in FindUnreferencedIdentifiers.java
6.  Add defined constants in Constants.java
7.  Reduce test resources to zero size to reduce git repository.  Modify file_size and checksum tags to reflect new parameters.

Refs:

#310 Validate missing collections in bundle after CCB-282 updates
jordanpadams added a commit that referenced this issue Apr 3, 2021
@jordanpadams
Merge pull request #320 from NASA-PDS/#310
7baa388 
Allows the name of a collection to not require an underscore immediat…
@jordanpadams jordanpadams mentioned this issue Aug 16, 2021
Closed
14 tasks
@jpl-jengelke
Copy link
Contributor

@jordanpadams During testing the underscore error doesn't show up but the test data doesn't pass, either:
310_execution.txt
test_310-1.txt
test_310-2.txt

Is this OK?

@jordanpadams
Copy link
Member Author

@jpl-jengelke we had a bug filed for this 1 day after you posted this comment :-) #408 . Can you try this again with the latest tagged version of validate?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@qchaupds qchaupds

Labels
B12.0 bug Something isn't working s.medium
Projects
None yet
Milestone
06.Mary.Decker.Slaney
Development

Successfully merging a pull request may close this issue.

Allows the name of a collection to not require an underscore immediat… NASA-PDS/validate
4 participants
@jordanpadams @jpl-jengelke @tloubrieu-jpl @qchaupds