Standalone extractor integration #218
Conversation
…e-extractor-integration
…tandalone-extractor-integration
Codecov Report
@@ Coverage Diff @@
## master #218 +/- ##
==========================================
- Coverage 95.76% 95.62% -0.14%
==========================================
Files 345 272 -73
Lines 17787 15786 -2001
==========================================
- Hits 17034 15096 -1938
+ Misses 753 690 -63
Continue to review full report at Codecov.
|
…tandalone-extractor-integration
There was a problem hiding this comment.
fact_extractor issue must be fixed before merge: fkie-cad/fact_extractor#5
It also adds a useless "files" folder, that confuses me
| @@ -83,11 +83,11 @@ def file_is_empty(file_path): | |||
| Returns False otherwise | |||
| ''' | |||
| try: | |||
| if os.path.getsize(file_path) == 0: | |||
| if os.path.getsize(str(file_path)) == 0: | |||
There was a problem hiding this comment.
this is not an issue but more a general discussion. Should we patch all our helper functions to supoort Paht objects?
I would say yes.
There was a problem hiding this comment.
Indeed. But I'd like to do this in a separate branch to not overcrowd this already substantial one
src/unpacker/tar_repack.py
Outdated
| def _repack_extracted_files(extraction_dir, out_file_path): | ||
| with Popen('tar -C {} -cvzf {} .'.format(extraction_dir.name, out_file_path), shell=True, stdout=PIPE) as process: | ||
| def _repack_extracted_files(extraction_dir: Path, out_file_path: str) -> bytes: | ||
| with Popen('tar -C {} -cvzf {} .'.format(extraction_dir, out_file_path), shell=True, stdout=PIPE) as process: |
There was a problem hiding this comment.
we might better use the execute shell command helper function herre
There was a problem hiding this comment.
but, but, that's not really code I worked on 😢
| @@ -1,6 +1,5 @@ | |||
| import logging | |||
There was a problem hiding this comment.
get_faf_bin_dir() is no longer needed
There was a problem hiding this comment.
has been removed
…e-extractor-integration
…e-extractor-integration
…e-extractor-integration
|
Will look to add a memory config option for extraction containers |
…tandalone-extractor-integration
src/unpacker/unpack_base.py
Outdated
| if return_code != 0: | ||
| error = 'Failed to execute docker extractor with code {}:\n{}'.format(return_code, output) | ||
| logging.error(error) | ||
| raise RuntimeError(error) | ||
|
|
||
| self.change_owner_back_to_me(tmp_dir) | ||
| all_items = list(Path(tmp_dir, 'files').glob('**/*')) | ||
| return [item for item in all_items if not item.is_dir()] | ||
| return [item for item in safe_rglob(Path(tmp_dir, 'files')) if not item.is_dir()] | ||
|
|
||
| def change_owner_back_to_me(self, directory: str = None, permissions='u+r'): | ||
| with Popen('sudo chown -R {}:{} {}'.format(getuid(), getgid(), directory), shell=True, stdout=PIPE, stderr=PIPE) as pl: |
There was a problem hiding this comment.
Could you change this to use execute_shell_command helper function. This command does not work correctly, since any output on stderr overwrites stdout.
…tandalone-extractor-integration
weidenba
added
under review
For extractor CI see