Support importing new NIST NSRL RDS version 3 format

[LuizFerrazBrazil]

Dear Nassif, hello! How are you?

I would like your help about the use of the NIST/NSRL with the IPED (version >= 4.0.0). I read the https://github.com/sepinf-inc/IPED/wiki/User-Manual#hashes-database-version--400, but I yet have some doubts.

First, I downloaded the "RDS_2022.03.1_modern.zip (microcomputer applications from 2010 to present)" database. After, I updated this database with the SQL scripts published from NIST. Ok!

Questions:

a) Can I use directly in IPED the database NIST KFF (SQlite database)? Or first do I have to import that database to "iped-hashes.db"?

b) Considering that I need to import the database NIST (SQlite database) to "iped-hashes.db".
In Wiki, We have: "The main file "NSRLFile.txt" and the product file "NSRLProd.txt" must be present in the same folder to allow importing of this hash set."
What would the files "NSRLFile.txt" and "NSRLProd.txt". Sorry, I didn't understand that part.

[lfcnassif]

Seems NIST changed the NSRL format from TXT files (RDS v2) to a SQLITE database (RDS v3):
https://s3.amazonaws.com/rds.nsrl.nist.gov/RDS/RDSv3_Docs/RDSv3.pdf

It was a recent change and it is still not supported by us (actually I just discovered it now).

I'm updating the title to track adding the new support to RDS v3.

[LuizFerrazBrazil]

Perfect! So, now, I will use the RDS v2.
Thanks for the quick answer!

[lfcnassif]

From PDF above:

With the RDSv3 publication, it will also be possible for users to construct an RDSv2 like
publication from data included in the RDSv3 format. Data in the four core NSRL files in RDSv2,
NSRLFile.txt, NSRLMfg.txt, NSRLOS.txt, and NSRLProd.txt, will be stored in a set of four VIEWS in
the RDSv3 SQLite publication, known as FILE, MFG, OS, and PKG (these views are defined bellow
in the included RDSv3 database schema). The NSRL plans to provide a method for users to
convert an RDSv3 publication into an RDSv2 like publication, for those who are interested

So it seems quite easy to convert RDS v3 to old v2 format and import the newer NSRL hashset into the tool before we implement direct support for it.

[wladimirleite]

It was a recent change and it is still not supported by us (actually I just discovered it now).
I'm updating the title to track adding the new support to RDS v3.

I was aware about this change.
It was announced a few months ago and currently the same data sets are available in the usual (v2) format.
From March/2023 only the SQLite (v3) format will be available, so there is still some time to implement this.

I can work in this issue.
My idea would be adding the NSRL SQLite as another format that can be imported in IPED's hashes DB.
Accessing it directly would be another possible solution, but I think it goes in the opposite direction of having a unified hashes DB.

[lfcnassif]

Thank you @tc-wleite! I also think this is not urgent.

My idea would be adding the NSRL SQLite as another format that can be imported in IPED's hashes DB.

I totally agree.

currently the same data sets are available in the usual (v2) format.

Just found them now. @Winiciusf you can download the v2 format from UDF images of RDS 2.79 CDs section at https://www.nist.gov/itl/ssd/software-quality-group/national-software-reference-library-nsrl/nsrl-download/current-rds

[LuizFerrazBrazil]

So it seems quite easy to convert RDS v3 to old v2 format and import the newer NSRL hashset into the tool before we implement direct support for it.

Great! I will try to perform this conversion this week.

[wladimirleite]

Great! I will try to perform this conversion this week.

Hi @Winiciusf!
Not sure if it was already clear, but you don't need to do any conversion.
Just download and import the NSRL hash sets in the usual (v2) format:
https://s3.amazonaws.com/rds.nsrl.nist.gov/RDS/current/RDS_modern.iso
https://s3.amazonaws.com/rds.nsrl.nist.gov/RDS/current/RDS_android.iso
https://s3.amazonaws.com/rds.nsrl.nist.gov/RDS/current/RDS_ios.iso

[LuizFerrazBrazil]

Ok! Thank you!Enviado do meu Galaxy -------- Mensagem original --------De : Wladimir Leite ***@***.***> Data: 22/12/2022 15:12 (GMT-03:00) Para: sepinf-inc/IPED ***@***.***> Cc: Winicius Ferraz Neres ***@***.***>, Mention ***@***.***> Assunto: Re: [sepinf-inc/IPED] Support importing new NIST NSRL RDS version 3 format (Issue #1461) Great! I will try to perform this conversion this week. Hi @Winiciusf! Not sure if it was already clear, but you don't need to do any conversion. Just download and import the NSRL hash sets in the usual (v2) format: https://s3.amazonaws.com/rds.nsrl.nist.gov/RDS/current/RDS_modern.iso https://s3.amazonaws.com/rds.nsrl.nist.gov/RDS/current/RDS_android.iso https://s3.amazonaws.com/rds.nsrl.nist.gov/RDS/current/RDS_ios.iso —Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>