KeyError: 'datadirectories'

[ashishcse0031]

`data_dir = "/home/cse31/MalReserach/data/ember/"

ember.create_vectorized_features(data_dir)
_ = ember.create_metadata(data_dir)
`
Vectorizing training set
0%| | 0/900000 [00:00<?, ?it/s]

RemoteTraceback Traceback (most recent call last)
RemoteTraceback:
"""
Traceback (most recent call last):
File "/home/cse31/anaconda3/lib/python3.7/multiprocessing/pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "/home/cse31/anaconda3/lib/python3.7/site-packages/ember-0.1.0-py3.7.egg/ember/init.py", line 44, in vectorize_unpack
return vectorize(*args)
File "/home/cse31/anaconda3/lib/python3.7/site-packages/ember-0.1.0-py3.7.egg/ember/init.py", line 31, in vectorize
feature_vector = extractor.process_raw_features(raw_features)
File "/home/cse31/anaconda3/lib/python3.7/site-packages/ember-0.1.0-py3.7.egg/ember/features.py", line 522, in process_raw_features
feature_vectors = [fe.process_raw_features(raw_obj[fe.name]) for fe in self.features]
File "/home/cse31/anaconda3/lib/python3.7/site-packages/ember-0.1.0-py3.7.egg/ember/features.py", line 522, in
feature_vectors = [fe.process_raw_features(raw_obj[fe.name]) for fe in self.features]
KeyError: 'datadirectories'
"""

The above exception was the direct cause of the following exception:

KeyError Traceback (most recent call last)
in
1 data_dir = "/home/cse31/MalReserach/data/ember/" # change this to where you unzipped the download
2
----> 3 ember.create_vectorized_features(data_dir)
4 _ = ember.create_metadata(data_dir)

~/anaconda3/lib/python3.7/site-packages/ember-0.1.0-py3.7.egg/ember/init.py in create_vectorized_features(data_dir, feature_version)
73 raw_feature_paths = [os.path.join(data_dir, "train_features_{}.jsonl".format(i)) for i in range(6)]
74 nrows = sum([1 for fp in raw_feature_paths for line in open(fp)])
---> 75 vectorize_subset(X_path, y_path, raw_feature_paths, extractor, nrows)
76
77 print("Vectorizing test set")

~/anaconda3/lib/python3.7/site-packages/ember-0.1.0-py3.7.egg/ember/init.py in vectorize_subset(X_path, y_path, raw_feature_paths, extractor, nrows)
58 argument_iterator = ((irow, raw_features_string, X_path, y_path, extractor, nrows)
59 for irow, raw_features_string in enumerate(raw_feature_iterator(raw_feature_paths)))
---> 60 for _ in tqdm.tqdm(pool.imap_unordered(vectorize_unpack, argument_iterator), total=nrows):
61 pass
62

~/anaconda3/lib/python3.7/site-packages/tqdm/_tqdm.py in iter(self)
1003 """), fp_write=getattr(self.fp, 'write', sys.stderr.write))
1004
-> 1005 for obj in iterable:
1006 yield obj
1007 # Update and possibly print the progressbar.

~/anaconda3/lib/python3.7/multiprocessing/pool.py in next(self, timeout)
746 if success:
747 return value
--> 748 raise value
749
750 next = next # XXX

KeyError: 'datadirectories'

[mrphilroth]

Are you attempting to run on the original dataset release containing samples from 2017? If so, then you'll need to specify the feature version number (in this case 1) when you vectorize the features:

ember.create_vectorized_features(data_dir, 1)

If you're working on the most recent dataset release, then there's definitely a bug I'll have to track down.

[ashishcse0031]

I was working with 2017 dataset, and it solved the issue.