Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Paper: Expert RF Feature Extraction to Win the Army RCO AI Signal Classification Challenge #472

Open
wants to merge 4 commits into
base: 2019
from

Conversation

Projects
None yet
5 participants
@Teque5
Copy link

commented May 22, 2019

This pull request is for 1of2 papers from The Aerospace Corporation approved for submission to the 2019 SciPy Conference.

OTR201900733 for the paper Expert RF Feature Extraction to Win the Army RCO
AI Signal Classification Challenge

Everything built OK on my box.

Expert RF Feature Extraction Paper
The following squashed commit was approved for public release by **The
Aerospace Corporation** on 2019-05-15. Commits made by the *Digital
Communications Implementation Department* are approved for public
release under request #OTR201900665 by the Office of Technical relations.
@deniederhut

This comment has been minimized.

Copy link
Member

commented May 22, 2019

Hm... I wonder if the server is mad about the slash in the branch name. Let me look into this.

@deniederhut

This comment has been minimized.

Copy link
Member

commented May 23, 2019

Looks like that did the trick! You should be able to see your papers on the build server now.

@thedatalass

This comment has been minimized.

Copy link

commented May 28, 2019

Paper reviewed by LKahn 5/28 and ready for conference proceedings

@yingweiy

This comment has been minimized.

Copy link

commented Jun 6, 2019

REVIEW COMMENTS

In this paper, the authors presented a method to predict the radio signal modulation type and SNR, which won the Army RCO AI signal classification challenge. This paper is well written. The method is technically sound in general, and the results appear to be reproducible. The data source are clearly referenced.

Here are my questions/suggestions:

  1. Is the source code available to the readers? This is required in the review criteria.
  2. Figure 1. "cv" refers to "cross validation". It is better add this notation in the figure caption.
  3. Eq 1 in page 2: the log loss basically is the "cross entropy", and this needs to be properly cited.
  4. The "shallow machine learning classifiers" are defined as "classifiers not build features out of the raw data". However, the deep learner as proposed also takes in the engineered features instead of deriving these features from the raw data. Is it the best practice? What if the DL algorithm directly applied to the raw data?
@Teque5

This comment has been minimized.

Copy link
Author

commented Jun 10, 2019

Is the source code available to the readers? This is required in the review criteria.

Releasing our entire codebase won't be possible. Instead I've tried to show which modules we relied upon like allantools and scipy.signal. I've also tried to make as clear as possible how individual features were calculated in latex. Moreover, since there really isn't an 'algorithm' at work here the code is really constrained to the individual features discussed. We may include code snippets on the poster that will be shown at the scipy 2019. Unfortunately getting code cleared for public release is quite difficult due to export control.

Figure 1. "cv" refers to "cross validation". It is better add this notation in the figure caption.

Ok no problem.

Eq 1 in page 2: the log loss basically is the "cross entropy", and this needs to be properly cited.

Ok; I can add this as a footnote.

The "shallow machine learning classifiers" are defined as "classifiers not build features out of the raw data". However, the deep learner as proposed also takes in the engineered features instead of deriving these features from the raw data. Is it the best practice? What if the DL algorithm directly applied to the raw data?

This is addressed in our companion paper, available here as a separate submission. The DL network produced better results than the engineering features, but combined (ensemble) yielded even better results. Moreover the engineering features based network was very robust against test data that is unlike the training data. A pure DL network stumbled in this regard.

@yingweiy If this is OK, let me know and I'll make the two changes you suggest.

@bbotran

This comment has been minimized.

Copy link

commented Jun 12, 2019

The paper is very well written and it is easy to follow. Signal processing is not my domain and I am not up to date with latest methodologies used in the field. However, the methodology is thorough and well explained. Even without releasing the code, I would definitely accept this paper.
I found a small issue in page 2, where the formula for recall and precision is the same one. Please, correct it.

@Teque5

This comment has been minimized.

Copy link
Author

commented Jun 12, 2019

@bbotran The formulas look similar, but are different. I believe the current version is correct.

@yingweiy

This comment has been minimized.

Copy link

commented Jun 12, 2019

@Teque5 Thanks for your reply. Yes, I can accept this paper with the two minor changes as suggested. Thanks!

@bbotran

This comment has been minimized.

Copy link

commented Jun 12, 2019

@bbotran The formulas look similar, but are different. I believe the current version is correct.

@Teque5 You are right. Ignore my comment. Congrats for your work. I recommend accepting the paper.

@Teque5

This comment has been minimized.

Copy link
Author

commented Jun 12, 2019

I just pushed the latest with the above changes as discussed.

@deniederhut

This comment has been minimized.

Copy link
Member

commented Jun 13, 2019

It looks like at least one of these papers is missing its DOI. Could I ask you to take a stroll through you bibliography and add these?

@Teque5

This comment has been minimized.

Copy link
Author

commented Jun 17, 2019

@deniederhut DOIs have been added/updated. Good catch.

@deniederhut

This comment has been minimized.

Copy link
Member

commented Jun 18, 2019

This looks good to go. Thanks so much for submitting!

@deniederhut deniederhut added ready and removed pending-comment labels Jun 18, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.