Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New sort_db is broken #1236

Closed
litghost opened this issue Feb 14, 2020 · 18 comments
Closed

New sort_db is broken #1236

litghost opened this issue Feb 14, 2020 · 18 comments
Assignees

Comments

@litghost
Copy link
Contributor

@litghost litghost commented Feb 14, 2020

  File "../../utils/sort_db.py", line 463, in <module>
    sys.exit(main(sys.argv))
  File "../../utils/sort_db.py", line 458, in main
    sort_dir('.')
  File "../../utils/sort_db.py", line 450, in sort_dir
    sort_file(n)
  File "../../utils/sort_db.py", line 422, in sort_file
    x = sort_json(n)
  File "../../utils/sort_db.py", line 385, in sort_json
    xjson.pprint(f, d)
  File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-artix7/utils/xjson.py", line 125, in pprint
    data = sort(data)
  File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-artix7/utils/xjson.py", line 117, in sort
    return rsorter(data)
  File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-artix7/utils/xjson.py", line 82, in rsorter
    nitems.append((key(k), k, rsorter(v)))
  File "/tmpfs/src/github/symbiflow-prjxray-presubmit-db-artix7/utils/xjson.py", line 108, in rsorter
    nlist.sort(key=lambda n: n[0])
TypeError: unorderable types: tuple() < NoneType()
@mithro
Copy link
Member

@mithro mithro commented Feb 14, 2020

Any idea what the input to xjson.py was there?

@litghost
Copy link
Contributor Author

@litghost litghost commented Feb 14, 2020

Doesn't matter?

@litghost
Copy link
Contributor Author

@litghost litghost commented Feb 14, 2020

I'll also point out that sorting ordered constructions (tuples and list) can break the data?

@mithro
Copy link
Member

@mithro mithro commented Feb 14, 2020

In Python 3, you can't sort something which has a None object in it...

In [1]: ['a', None, '1'].sort()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-46c34c36c41c> in <module>()
----> 1 ['a', None, '1'].sort()

TypeError: '<' not supported between instances of 'NoneType' and 'str'
@litghost
Copy link
Contributor Author

@litghost litghost commented Feb 14, 2020

In Python 3, you can't sort something which has a None object in it...

In [1]: ['a', None, '1'].sort()
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-1-46c34c36c41c> in <module>()
----> 1 ['a', None, '1'].sort()

TypeError: '<' not supported between instances of 'NoneType' and 'str'

And? What's you point? Your code is attempting to modified already ordered data? That's not a valid thing to do?

@litghost
Copy link
Contributor Author

@litghost litghost commented Feb 14, 2020

To be clear:
{'a': 1, 'b': 2} is the same as {'b': 2, 'a': 1}, but [1, 3, 2] is not the same thing as [1, 2, 3].

@mithro
Copy link
Member

@mithro mithro commented Feb 14, 2020

The issue is that JSON only has arrays and dicts. In our JSON data structure, almost all our lists (that are not 2 in length) should be sets (and set([1, 3, 2]) is the same thing as set([1, 2, 3])). Are there examples where this is not the case?

@litghost
Copy link
Contributor Author

@litghost litghost commented Feb 14, 2020

Almost every list in the JSON is ordered. For example, the timing data is defined as the following tuple: https://github.com/SymbiFlow/prjxray/blob/master/prjxray/timing.py#L162

GitHub
Documenting the Xilinx 7-series bit-stream format. - SymbiFlow/prjxray
@litghost
Copy link
Contributor Author

@litghost litghost commented Feb 14, 2020

almost all our lists should be sets

That just isn't true?

@mithro
Copy link
Member

@mithro mithro commented Feb 14, 2020

Some examples;

  • tileconn.json's top level
  • tileconn.json: XXX['wire_paris']
  • design.json: XXX['ports']
  • design.json: XXX['ports']['wires_outside_roi']
  • site_type_XXXX.json - No lists
  • tile_type_XXXX['sites']
@litghost
Copy link
Contributor Author

@litghost litghost commented Feb 14, 2020

Some examples;

  • tileconn.json's top level
  • tileconn.json: XXX['wire_paris']
  • design.json: XXX['ports']
  • design.json: XXX['ports']['wires_outside_roi']
  • site_type_XXXX.json - No lists
  • tile_type_XXXX['sites']

And I already provided a counter example? You cannot just start reordering ordered data and expect it to work? Your code was broken.

@mithro
Copy link
Member

@mithro mithro commented Feb 14, 2020

We have two options;

  • Special case the ordered data.
  • Special case the unordered data.

The ordered data is currently, lists of wire pairs/coordinates and the timing data? Are there others?

@litghost
Copy link
Contributor Author

@litghost litghost commented Feb 14, 2020

Or just don't guess and randomly break things?

@mithro
Copy link
Member

@mithro mithro commented Feb 14, 2020

Inside the JSON [] objects in our current database we current have both;

  • (a) data which should be considered ordered, and
  • (b) data which should be considered unordered.

We can thus;

  • (1) Define [] as ordered and convert the currently unordered data into a different structure.
  • (2) Define [] as unordered and convert the currently ordered data into a different structure.

I believe you wish to do (1) -- correct?

@litghost
Copy link
Contributor Author

@litghost litghost commented Feb 14, 2020

Neither: Define [] as always ordered. Fuzzers that write [] should have a stable order.

@litghost
Copy link
Contributor Author

@litghost litghost commented Feb 14, 2020

  • tileconn.json: XXX['wire_paris']

This is an example of ordered data, not unordered data.

@mithro
Copy link
Member

@mithro mithro commented Feb 14, 2020

The pairs inside XXX['wire_pairs'] are ordered but the actual XXX['wire_pairs'] list itself is unordered?

@litghost
Copy link
Contributor Author

@litghost litghost commented Feb 14, 2020

The pairs inside XXX['wire_pairs'] are ordered but the actual XXX['wire_pairs'] list itself is unordered?

True

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

2 participants
You can’t perform that action at this time.