-
Notifications
You must be signed in to change notification settings - Fork 68
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Graph - reuse parse content dto, skip empty packs #4154
Merged
Merged
Changes from 16 commits
Commits
Show all changes
24 commits
Select commit
Hold shift + click to select a range
be57b2d
test
YuvHayun e9e6c9b
tests
YuvHayun d989017
tests
YuvHayun 9663e46
cache content dto
994bbd5
delete maxsize
e0d811e
dont modify env var
0ad1768
fix packs to parse
b2d7718
skip invalid
623be9d
clear cache in tests
67cc80a
remove cache in test suite
12890ab
fix tests
24a4bba
CONTENT_PATH
38d3af5
common
2a52607
remove lru cache from class
94e288e
docs
e8a3333
fix cache clear
9d6121d
Update demisto_sdk/commands/content_graph/parsers/repository.py
ilaner 4e376ab
Update demisto_sdk/commands/content_graph/objects/repository.py
ilaner 85db15b
CR Notes
df72853
typing fix
9e8c743
Merge branch 'skip_graph_validation_on_old_validate' of github.com:de…
3bfd3e5
cache clear
71676e7
Merge remote-tracking branch 'origin/master' into skip_graph_validati…
84c199c
update
File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,9 @@ | ||
import shutil | ||
import time | ||
from functools import lru_cache | ||
from multiprocessing.pool import Pool | ||
from pathlib import Path | ||
from typing import List, Optional | ||
from typing import List, Optional, Tuple | ||
|
||
import tqdm | ||
from pydantic import BaseModel, DirectoryPath | ||
|
@@ -17,25 +18,41 @@ | |
USE_MULTIPROCESSING = False # toggle this for better debugging | ||
|
||
|
||
@lru_cache | ||
def from_path(path: Path = CONTENT_PATH, packs_to_parse: Optional[Tuple[str]] = None): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. why isn't it a class method of There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I documented, for some reason I get cache miss whin it's in the class. |
||
""" | ||
Returns a ContentDTO object with all the packs of the content repository. | ||
|
||
This function is outside of the class for better caching. | ||
The class function uses this function so the behavior is the same. | ||
""" | ||
repo_parser = RepositoryParser(path) | ||
with tqdm.tqdm( | ||
total=len(tuple(repo_parser.iter_packs())) | ||
if not packs_to_parse | ||
else len(packs_to_parse), | ||
unit="packs", | ||
desc="Parsing packs", | ||
position=0, | ||
leave=True, | ||
) as progress_bar: | ||
repo_parser.parse(progress_bar=progress_bar) | ||
content_dto = ContentDTO.from_orm(repo_parser) | ||
return content_dto | ||
ilaner marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
|
||
class ContentDTO(BaseModel): | ||
path: DirectoryPath = Path(CONTENT_PATH) # type: ignore | ||
packs: List[Pack] | ||
|
||
@staticmethod | ||
def from_path(path: Path = CONTENT_PATH): | ||
def from_path( | ||
path: Path = CONTENT_PATH, packs_to_parse: Optional[Tuple[str]] = None | ||
): | ||
""" | ||
Returns a ContentDTO object with all the packs of the content repository. | ||
""" | ||
repo_parser = RepositoryParser(path) | ||
with tqdm.tqdm( | ||
total=len(tuple(repo_parser.iter_packs())), | ||
unit="packs", | ||
desc="Parsing packs", | ||
position=0, | ||
leave=True, | ||
) as progress_bar: | ||
repo_parser.parse(progress_bar=progress_bar) | ||
return ContentDTO.from_orm(repo_parser) | ||
return from_path(path, packs_to_parse) | ||
|
||
def dump( | ||
self, | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
put inside a variable and reuse it also in the line 76