New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

LibrePCB file format discussion #150

Closed
szg0000 opened this Issue Feb 16, 2017 · 19 comments

Comments

Projects
None yet
4 participants
@szg0000

szg0000 commented Feb 16, 2017

In this issue we discuss the LibrePCB supported file formats. Feel free to comment.
First, we need a document which describes the latest version of LibrePCB file formats.

@szg0000

This comment has been minimized.

Show comment
Hide comment
@szg0000

szg0000 Feb 20, 2017

I see SPICE modelling is under development. Can we invite SPICE developers into LibrePCB project?

I haven't seen any gerber file format description in the documentation. Can we invite linuxcnc.org and http://gerbv.geda-project.org/ developers into LibrePCB project?

szg0000 commented Feb 20, 2017

I see SPICE modelling is under development. Can we invite SPICE developers into LibrePCB project?

I haven't seen any gerber file format description in the documentation. Can we invite linuxcnc.org and http://gerbv.geda-project.org/ developers into LibrePCB project?

@dbrgn

This comment has been minimized.

Show comment
Hide comment
@dbrgn

dbrgn Feb 20, 2017

Contributor

Gerber file format should be Gerber X2 (can be found on the website of Ucamco).

I started work on https://github.com/dbrgn/gerber-types-rs. If C++ integration with Rust improves this year (it's on the roadmap) we might be able to integrate the library.

Contributor

dbrgn commented Feb 20, 2017

Gerber file format should be Gerber X2 (can be found on the website of Ucamco).

I started work on https://github.com/dbrgn/gerber-types-rs. If C++ integration with Rust improves this year (it's on the roadmap) we might be able to integrate the library.

@szg0000

This comment has been minimized.

Show comment
Hide comment
@szg0000

szg0000 Feb 20, 2017

OK.
What about the UTF-8 support? In the file path, file names, and on the board layer texts, schematic layer texts?

szg0000 commented Feb 20, 2017

OK.
What about the UTF-8 support? In the file path, file names, and on the board layer texts, schematic layer texts?

@dbrgn

This comment has been minimized.

Show comment
Hide comment
@dbrgn

dbrgn Feb 20, 2017

Contributor

As far as I know this should be covered by QT. @ubruhin probably knows more.

Contributor

dbrgn commented Feb 20, 2017

As far as I know this should be covered by QT. @ubruhin probably knows more.

@ubruhin

This comment has been minimized.

Show comment
Hide comment
@ubruhin

ubruhin Feb 20, 2017

Member

As far as I know this should be covered by QT.

Right, basically UTF-8 is supported everywhere in the application.

Member

ubruhin commented Feb 20, 2017

As far as I know this should be covered by QT.

Right, basically UTF-8 is supported everywhere in the application.

@dbrgn

This comment has been minimized.

Show comment
Hide comment
@dbrgn
Contributor

dbrgn commented Jun 9, 2017

@ubruhin

This comment has been minimized.

Show comment
Hide comment
Member

ubruhin commented Jun 11, 2017

@dbrgn

This comment has been minimized.

Show comment
Hide comment
@dbrgn

dbrgn Jun 11, 2017

Contributor

Wow, I really like the general look of the files.

Something I noticed... Usually you would translate this:

<author>LibrePCB</author>

into this:

(author "LibrePCB")

But in this case:

<keywords locale="en_US">resistor,resistance</keywords>

it becomes

(keywords (locale "en_US") (value "resistor,resistance"))

The "value" expression is not really necessary, you could convert it to:

(keywords (locale "en_US") "resistor,resistance")

Then, while parsing, if you get an expression you can treat it as an attribute, while if you get a "plain" value you can treat it as the value.

Another example:

(pin (length 2.000000) (rotation 180.000000) (uuid "2b3dd7f8-043b-4d43-9302-9300ba356de7") (x 5.080000) (y 0.000000) (value 2))

vs

(pin (length 2.000000) (rotation 180.000000) (uuid "2b3dd7f8-043b-4d43-9302-9300ba356de7") (x 5.080000) (y 0.000000) 2)
Contributor

dbrgn commented Jun 11, 2017

Wow, I really like the general look of the files.

Something I noticed... Usually you would translate this:

<author>LibrePCB</author>

into this:

(author "LibrePCB")

But in this case:

<keywords locale="en_US">resistor,resistance</keywords>

it becomes

(keywords (locale "en_US") (value "resistor,resistance"))

The "value" expression is not really necessary, you could convert it to:

(keywords (locale "en_US") "resistor,resistance")

Then, while parsing, if you get an expression you can treat it as an attribute, while if you get a "plain" value you can treat it as the value.

Another example:

(pin (length 2.000000) (rotation 180.000000) (uuid "2b3dd7f8-043b-4d43-9302-9300ba356de7") (x 5.080000) (y 0.000000) (value 2))

vs

(pin (length 2.000000) (rotation 180.000000) (uuid "2b3dd7f8-043b-4d43-9302-9300ba356de7") (x 5.080000) (y 0.000000) 2)
@ubruhin

This comment has been minimized.

Show comment
Hide comment
@ubruhin

ubruhin Jun 11, 2017

Member

Of course there is room for improvements, I just modified a few lines of code to quick&dirty get some s-expressions files. But if we really decide to switch from xml to s-expressions, there is lot more work to do 😉

Member

ubruhin commented Jun 11, 2017

Of course there is room for improvements, I just modified a few lines of code to quick&dirty get some s-expressions files. But if we really decide to switch from xml to s-expressions, there is lot more work to do 😉

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Jun 15, 2017

Some googling around file formats used in different EDA leaded to this these links :

  1. EDIF file format using s expressions : https://en.wikipedia.org/wiki/EDIF
  2. IC description : https://en.wikipedia.org/wiki/Caltech_Intermediate_Form
  3. IC related db : https://en.wikipedia.org/wiki/GDSII
  4. OASIS industry standard : https://en.wikipedia.org/wiki/Open_Artwork_System_Interchange_Standard
  5. Data Exchange with ISO 10303 : http://stepcode.org/
  6. PADS specification from Mentor Graphics : ftp://ftp.freecalypso.org/pub/CAD/PADS/pdfdocs/Plib_ASCII.pdf / ftp://ftp.freecalypso.org/pub/CAD/PADS/pdfdocs/Ppcb_ASCII.pdf
  7. KiCAD file spec : http://bazaar.launchpad.net/~stambaughw/kicad/doc-read-only/download/head:/1115%4016bec504-3128-0410-b3e8-8e38c2123bca:trunk%252Fkicad-doc%252Fdoc%252Fhelp%252Ffile_formats%252Ffile_formats.pdf/file_formats.pdf

The very intersting Qucs project (http://qucs.sourceforge.net/screenshots.html) has an interesting discussion around the subject : Qucs/qucs#475

The best would be to take the best from each to make the best blend! IMO We should developp a custom file format , ASCII readable , based upon maybe Json or S-expr for encapsulating data . I also like very much STEPCODE because of ISO compliance. In general shape the main project file should have an header regarding with project properties, general vars etc... and references to the project subparts . a json / s expr parser is far more lighter than the xml one too.

ghost commented Jun 15, 2017

Some googling around file formats used in different EDA leaded to this these links :

  1. EDIF file format using s expressions : https://en.wikipedia.org/wiki/EDIF
  2. IC description : https://en.wikipedia.org/wiki/Caltech_Intermediate_Form
  3. IC related db : https://en.wikipedia.org/wiki/GDSII
  4. OASIS industry standard : https://en.wikipedia.org/wiki/Open_Artwork_System_Interchange_Standard
  5. Data Exchange with ISO 10303 : http://stepcode.org/
  6. PADS specification from Mentor Graphics : ftp://ftp.freecalypso.org/pub/CAD/PADS/pdfdocs/Plib_ASCII.pdf / ftp://ftp.freecalypso.org/pub/CAD/PADS/pdfdocs/Ppcb_ASCII.pdf
  7. KiCAD file spec : http://bazaar.launchpad.net/~stambaughw/kicad/doc-read-only/download/head:/1115%4016bec504-3128-0410-b3e8-8e38c2123bca:trunk%252Fkicad-doc%252Fdoc%252Fhelp%252Ffile_formats%252Ffile_formats.pdf/file_formats.pdf

The very intersting Qucs project (http://qucs.sourceforge.net/screenshots.html) has an interesting discussion around the subject : Qucs/qucs#475

The best would be to take the best from each to make the best blend! IMO We should developp a custom file format , ASCII readable , based upon maybe Json or S-expr for encapsulating data . I also like very much STEPCODE because of ISO compliance. In general shape the main project file should have an header regarding with project properties, general vars etc... and references to the project subparts . a json / s expr parser is far more lighter than the xml one too.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Jun 15, 2017

parsing s expressions : http://people.csail.mit.edu/rivest/sexp.html
IETF draft about s expressions (the most standard ref about it) : http://people.csail.mit.edu/rivest/Sexp.txt

ghost commented Jun 15, 2017

parsing s expressions : http://people.csail.mit.edu/rivest/sexp.html
IETF draft about s expressions (the most standard ref about it) : http://people.csail.mit.edu/rivest/Sexp.txt

@dbrgn

This comment has been minimized.

Show comment
Hide comment
@dbrgn

dbrgn Jun 15, 2017

Contributor

Following that IETF draft might be a good idea. In that case using the reference C implementation might save us some time.

Alternatively if we want to do our own stuff we could also use a library like sexpresso.

Contributor

dbrgn commented Jun 15, 2017

Following that IETF draft might be a good idea. In that case using the reference C implementation might save us some time.

Alternatively if we want to do our own stuff we could also use a library like sexpresso.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Jun 15, 2017

We also can go for tweaking the original rivest C implementation . The advanced form sounds the most usable and readable

ghost commented Jun 15, 2017

We also can go for tweaking the original rivest C implementation . The advanced form sounds the most usable and readable

@ubruhin

This comment has been minimized.

Show comment
Hide comment
@ubruhin

ubruhin Oct 10, 2017

Member

So I would say LibrePCB should completely switch from XML to S-Expressions because that file format is very simple and seems to provide all the functionality which LibrePCB needs. Or are there any other opinions?

If everyone agrees with switching to S-expressions, I would close this issue and open a new one to track the progress of the file format change and to discuss more details (there will be some details to be clarified).

Member

ubruhin commented Oct 10, 2017

So I would say LibrePCB should completely switch from XML to S-Expressions because that file format is very simple and seems to provide all the functionality which LibrePCB needs. Or are there any other opinions?

If everyone agrees with switching to S-expressions, I would close this issue and open a new one to track the progress of the file format change and to discuss more details (there will be some details to be clarified).

@dbrgn

This comment has been minimized.

Show comment
Hide comment
@dbrgn

dbrgn Oct 10, 2017

Contributor

Awesome! I agree :)

Contributor

dbrgn commented Oct 10, 2017

Awesome! I agree :)

@itviewer

This comment has been minimized.

Show comment
Hide comment
@itviewer

itviewer Oct 11, 2017

why not json?
I think very few people are familiar with S-expressions ,while json is everywhere.
We are developing an EDA tool and decided to use json format,with this https://github.com/nlohmann/json as parser,so far,the conversion of json and Qt data types is very straightforward.

itviewer commented Oct 11, 2017

why not json?
I think very few people are familiar with S-expressions ,while json is everywhere.
We are developing an EDA tool and decided to use json format,with this https://github.com/nlohmann/json as parser,so far,the conversion of json and Qt data types is very straightforward.

@ubruhin

This comment has been minimized.

Show comment
Hide comment
@ubruhin

ubruhin Oct 11, 2017

Member

why not json?

Basically because S-Expressions are more compact and readable. Consider this LibrePCB Symbol definition in S-Expressions:

(symbol
 (uuid "00000000-0000-4001-8000-000000000000")
 (version "0.1")
 (author "LibrePCB")
 (name (locale "") "foo")
 (name (locale "gsw_CH") "bar")
 (description (locale "")" foobar")
 (description (locale "gsw_CH") "barfoo")
 (category "00000000-0000-4001-8000-000000000000")
 (pin (uuid "702df32a-635f-46e9-85d2-5b7e2c20aa87") (x -17.78) (y 2.54) (length 5.08) (rotation 0) "1")
 (pin (uuid "1339ffc8-afb2-4137-9d37-d9801ac2bfd0") (x -17.78) (y 0) (length 5.08) (rotation 0) "2")
 (pin (uuid "8207ee9b-89e5-41bc-a633-078952a6c370") (x -17.78) (y -2.54) (length 5.08) (rotation 0) "3")
 (pin (uuid "b5ec7a0f-0f26-482e-a68e-e9a8f108cb80") (x -17.78) (y -5.08) (length 5.08) (rotation 0) "4")
 (polygon (layer "sym_outlines") (width 0.254) (fill false) (grab_area true) (start_x -12.7) (start_y -7.62)
  (segment (angle 0) (end_x -12.7) (end_y 5.08))
  (segment (angle 0) (end_x 12.7) (end_y 5.08))
  (segment (angle 0) (end_x 12.7) (end_y -7.62))
  (segment (angle 0) (end_x -12.7) (end_y -7.62))
 )
 (text (layer "sym_names") (align_v "bottom") (height 2.54) (rotation 0) (x -12.7) (y 5.08) (align_h "left") "#NAME")
 (text (layer "sym_values") (align_v "top") (height 2.54) (rotation 0) (x -12.7) (y -7.62) (align_h "left") "#VALUE")
)

compared to JSON (with manually removed linebreaks to keep it shorter):

{
 "uuid": "00000000-0000-4001-8000-000000000000",
 "version": "0.1",
 "author": "LibrePCB",
 "name": {"": "foo", "gsw_CH": "bar"},
 "description": {"": "foobar", "gsw_CH": "barfoo"},
 "category": ["00000000-0000-4001-8000-000000000000"],
 "pin": [
  {"uuid": "702df32a-635f-46e9-85d2-5b7e2c20aa87", "x": "-17.78", "y": "2.54", "length": "5.08", "rotation": "0", "value": "1"},
  {"uuid": "1339ffc8-afb2-4137-9d37-d9801ac2bfd0", "x": "-17.78", "y": "0", "length": "5.08", "rotation": "0", "value": "2"},
  {"uuid": "8207ee9b-89e5-41bc-a633-078952a6c370", "x": "-17.78", "y": "-2.54", "length": "5.08", "rotation": "0", "value": "3"},
  {"uuid": "b5ec7a0f-0f26-482e-a68e-e9a8f108cb80", "x": "-17.78", "y": "-5.08", "length": "5.08", "rotation": "0", "value": "4"}
 ],
 "polygon": [
  {
   "layer": "sym_outlines", "width": "0.254", "fill": "false", "grab_area": "true", "start_x": "-12.7", "start_y": "-7.62", "segment": [
   {"angle": "0", "end_x": "-12.7", "end_y": "5.08"},
   {"angle": "0", "end_x": "12.7", "end_y": "5.08"},
   {"angle": "0", "end_x": "12.7", "end_y": "-7.62"},
   {"angle": "0", "end_x": "-12.7", "end_y": "-7.62"}
  ]
 }],
 "text": [
  {"layer": "sym_names", "align_v": "bottom", "height": "2.54", "rotation": "0", "x": "-12.7", "y": "5.08", "align_h": "left", "value": "#NAME"},
  {"layer": "sym_values", "align_v": "top", "height": "2.54", "rotation": "0", "x": "-12.7", "y": "-7.62", "align_h": "left", "value": "#VALUE"}
 ]
}

The S-Expression variant is ~15% shorter (count of characters) and IMO also more readable.

Member

ubruhin commented Oct 11, 2017

why not json?

Basically because S-Expressions are more compact and readable. Consider this LibrePCB Symbol definition in S-Expressions:

(symbol
 (uuid "00000000-0000-4001-8000-000000000000")
 (version "0.1")
 (author "LibrePCB")
 (name (locale "") "foo")
 (name (locale "gsw_CH") "bar")
 (description (locale "")" foobar")
 (description (locale "gsw_CH") "barfoo")
 (category "00000000-0000-4001-8000-000000000000")
 (pin (uuid "702df32a-635f-46e9-85d2-5b7e2c20aa87") (x -17.78) (y 2.54) (length 5.08) (rotation 0) "1")
 (pin (uuid "1339ffc8-afb2-4137-9d37-d9801ac2bfd0") (x -17.78) (y 0) (length 5.08) (rotation 0) "2")
 (pin (uuid "8207ee9b-89e5-41bc-a633-078952a6c370") (x -17.78) (y -2.54) (length 5.08) (rotation 0) "3")
 (pin (uuid "b5ec7a0f-0f26-482e-a68e-e9a8f108cb80") (x -17.78) (y -5.08) (length 5.08) (rotation 0) "4")
 (polygon (layer "sym_outlines") (width 0.254) (fill false) (grab_area true) (start_x -12.7) (start_y -7.62)
  (segment (angle 0) (end_x -12.7) (end_y 5.08))
  (segment (angle 0) (end_x 12.7) (end_y 5.08))
  (segment (angle 0) (end_x 12.7) (end_y -7.62))
  (segment (angle 0) (end_x -12.7) (end_y -7.62))
 )
 (text (layer "sym_names") (align_v "bottom") (height 2.54) (rotation 0) (x -12.7) (y 5.08) (align_h "left") "#NAME")
 (text (layer "sym_values") (align_v "top") (height 2.54) (rotation 0) (x -12.7) (y -7.62) (align_h "left") "#VALUE")
)

compared to JSON (with manually removed linebreaks to keep it shorter):

{
 "uuid": "00000000-0000-4001-8000-000000000000",
 "version": "0.1",
 "author": "LibrePCB",
 "name": {"": "foo", "gsw_CH": "bar"},
 "description": {"": "foobar", "gsw_CH": "barfoo"},
 "category": ["00000000-0000-4001-8000-000000000000"],
 "pin": [
  {"uuid": "702df32a-635f-46e9-85d2-5b7e2c20aa87", "x": "-17.78", "y": "2.54", "length": "5.08", "rotation": "0", "value": "1"},
  {"uuid": "1339ffc8-afb2-4137-9d37-d9801ac2bfd0", "x": "-17.78", "y": "0", "length": "5.08", "rotation": "0", "value": "2"},
  {"uuid": "8207ee9b-89e5-41bc-a633-078952a6c370", "x": "-17.78", "y": "-2.54", "length": "5.08", "rotation": "0", "value": "3"},
  {"uuid": "b5ec7a0f-0f26-482e-a68e-e9a8f108cb80", "x": "-17.78", "y": "-5.08", "length": "5.08", "rotation": "0", "value": "4"}
 ],
 "polygon": [
  {
   "layer": "sym_outlines", "width": "0.254", "fill": "false", "grab_area": "true", "start_x": "-12.7", "start_y": "-7.62", "segment": [
   {"angle": "0", "end_x": "-12.7", "end_y": "5.08"},
   {"angle": "0", "end_x": "12.7", "end_y": "5.08"},
   {"angle": "0", "end_x": "12.7", "end_y": "-7.62"},
   {"angle": "0", "end_x": "-12.7", "end_y": "-7.62"}
  ]
 }],
 "text": [
  {"layer": "sym_names", "align_v": "bottom", "height": "2.54", "rotation": "0", "x": "-12.7", "y": "5.08", "align_h": "left", "value": "#NAME"},
  {"layer": "sym_values", "align_v": "top", "height": "2.54", "rotation": "0", "x": "-12.7", "y": "-7.62", "align_h": "left", "value": "#VALUE"}
 ]
}

The S-Expression variant is ~15% shorter (count of characters) and IMO also more readable.

@dbrgn

This comment has been minimized.

Show comment
Hide comment
@dbrgn

dbrgn Oct 11, 2017

Contributor

S-Expressions are also much simpler to parse. And with a custom generator (which is done in a few lines of code) we can make sure that fields are always written out in the same order, preventing noice in SCM. When switching JSON serializers, the order can change, sometimes even when sticking to the same library.

But I think that has already been discussed a few times :)

Contributor

dbrgn commented Oct 11, 2017

S-Expressions are also much simpler to parse. And with a custom generator (which is done in a few lines of code) we can make sure that fields are always written out in the same order, preventing noice in SCM. When switching JSON serializers, the order can change, sometimes even when sticking to the same library.

But I think that has already been discussed a few times :)

@ubruhin

This comment has been minimized.

Show comment
Hide comment
@ubruhin

ubruhin Oct 17, 2017

Member

Further discussion about S-Expressions moved to #192.

Member

ubruhin commented Oct 17, 2017

Further discussion about S-Expressions moved to #192.

@ubruhin ubruhin closed this Oct 17, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment