Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add the IX-F Member Export URL to the ixlan API endpoint #249

Closed
arnoldnipper opened this issue Nov 16, 2017 · 18 comments · Fixed by #762
Closed

Add the IX-F Member Export URL to the ixlan API endpoint #249

arnoldnipper opened this issue Nov 16, 2017 · 18 comments · Fixed by #762
Assignees
Labels
Milestone

Comments

@arnoldnipper
Copy link
Contributor

  1. if you want to automate and process, the API is the only way to do this

  2. all information which is in the user GUI should also be accessible via the API

@vegu vegu added the Time:Minor Up to 4 hours label Dec 13, 2017
@grizz grizz self-assigned this Nov 6, 2019
@grizz
Copy link
Member

grizz commented Nov 6, 2019

One of the problems with this is that some IXPs don't want the list being public I believe.

I'm for this if we also add a tick box for import_url_is_public or something.

@peeringdb/pc thoughts?

@arnoldnipper
Copy link
Contributor Author

+1 ... however, this information is public via "curl -sGL https://api.ixpdb.net/v1/provider/list | jq -r -c '.[] | select (.apis.ixfexport != "") | .apis.ixfexport'" atm

@shane-kerr
Copy link

So is the proposal to first to add a tick box specifying whether an IXP to allow listing members publicly, right? That creates several questions:

  • What should the default be on creation (I guess default to open to mimic current behavior, but opinions may vary)?
  • What should we set existing IXP to for this value (again, probably open)?
  • Do we need to somehow alert IXP that this is now an option? (A simple way would be contacting mentioning this in release notes, but that is basically a CYA approach. A better way would be e-mail. The "best" way might be to make a table containing all IXP that have not yet chosen a value and alerting them when they use the GUI.)

Then once this option is added, we have two additional tasks:

  1. Create an API call like requested in this ticket initially.
  2. Update the GUI to respect the new option.

Does that cover it?

@arnoldnipper
Copy link
Contributor Author

  • The default should be "no" to reflect the existing behaviour
  • for existing IX-F imports, visibility should be set to "no"
  • An email to announce should do. No need to contact IXes individually

@grizz
Copy link
Member

grizz commented Nov 7, 2019

+1 ... however, this information is public via "curl -sGL https://api.ixpdb.net/v1/provider/list | jq -r -c '.[] | select (.apis.ixfexport != "") | .apis.ixfexport'" atm

There may be a case that an IXP would share it privately with PDB, but still not expose it to the public. At least I think, am I wrong in that assumption?

@grizz
Copy link
Member

grizz commented Nov 7, 2019

  • The default should be "no" to reflect the existing behaviour
  • for existing IX-F imports, visibility should be set to "no"
  • An email to announce should do. No need to contact IXes individually

Agree.

Then once this option is added, we have two additional tasks:

  1. Create an API call like requested in this ticket initially.
  2. Update the GUI to respect the new option.

Does that cover it?

Yep, 1 will happen automatically just by creating the new the field, so it will just be 2.

@arnoldnipper
Copy link
Contributor Author

Would you mind to summarize and to call for voting, @grizz ... afais this issue has already support from @shane-kerr @grizz and @arnoldnipper

@grizz
Copy link
Member

grizz commented Nov 13, 2019

@arnoldnipper sure

  • An email to announce should do. No need to contact IXes individually

By this, you mean just a standard new version with X features to pdb-announce, right?

@arnoldnipper
Copy link
Contributor Author

By this, you mean just a standard new version with X features to pdb-announce, right?

Right, @grizz

@grizz
Copy link
Member

grizz commented Nov 13, 2019

Add two new fields to the API:

  • ixf_import_url string for the URL to fetch
  • ixf_import_url_visible string for visibility the same as the current poc records: public, users, private

ixf_import_url_visible will default to private which means it will only display for users who are members of the org

@peeringdb/pc slight change to the visibility to keep it in line with poc records

@arnoldnipper
Copy link
Contributor Author

+1 for what @grizz proposed

@grizz
Copy link
Member

grizz commented Apr 16, 2020

@peeringdb/pc need another vote please

@arnoldnipper
Copy link
Contributor Author

@grizz, I see support from you, @shane-kerr and me. That's three. How many do we need?

@ynbrthr
Copy link

ynbrthr commented Apr 16, 2020

here is another one: +1

@grizz grizz added this to Backlog in Data Ownership TF via automation Apr 18, 2020
@arnoldnipper arnoldnipper changed the title Add the IXF Member Export URL to the ixlan API endpoint Add the IX-F Member Export URL to the ixlan API endpoint Apr 22, 2020
@grizz grizz moved this from Backlog to TIME_WAIT in Data Ownership TF Apr 24, 2020
@grizz grizz moved this from TIME_WAIT to Ready for Implementation in Data Ownership TF Apr 25, 2020
@koalafil
Copy link

Can we pls have the Release Notes? Thx.

@koalafil
Copy link

@arnoldnipper Release Notes pls

@vegu
Copy link
Contributor

vegu commented Jul 13, 2020

This was specified to add the following fields ixf_import_url and ixf_import_url_visible

However the url field is already exposed on the API for POST and PUT requests as ixf_ixp_member_list_url, in order to allow people to set it.

For consistency the visibility field will be named ixf_ixp_member_list_url_visible as well.

In summary, the fields available on the api will be ixf_ixp_member_list_url and ixf_ixp_member_list_url_visible

@arnoldnipper
Copy link
Contributor Author

Release Notes
There are two new fields in the ixlan section resp. the ixlan object, called ixf_ixp_member_list_url and ixf_ixp_member_list_url_visible. The first one contains the URL to the IX-F import file while the second governs the visibility of the URL. Values are the same as for the poc object, namely "Public", "Users" and "Private", defaulting to "Private" (i.e. or users who are members of the org)

vegu added a commit to peeringdb/django-peeringdb that referenced this issue Jul 14, 2020
grizz added a commit to peeringdb/django-peeringdb that referenced this issue Jul 15, 2020
* add py3.7 tests and mark as supported version,
remove py3.4 tests and remove from support versions

* gettext_lazy

* setup kw

* tox - pass posargs

* Add ixf_last_import and ixf_net_count fields to ix object (peeringdb/peeringdb#683)

* Add the IX-F Member Export URL to the ixlan API endpoint (peeringdb/peeringdb#249)

* changelog

Co-authored-by: Stefan Pratter <stefan@20c.com>
Co-authored-by: Roy Crihfield <roy@20c.com>
vegu added a commit that referenced this issue Jul 15, 2020
vegu added a commit that referenced this issue Jul 15, 2020
Add the IX-F Member Export URL to the ixlan API endpoint (#249)

See merge request gh/peeringdb/peeringdb!92
@grizz grizz mentioned this issue Jul 15, 2020
Data Ownership TF automation moved this from Ready for Implementation to Done Jul 15, 2020
grizz added a commit that referenced this issue Jul 15, 2020
* Change label from primary ASN to ASN

* Raise validation error when trying to update ASN

* first steps for dotf importer procotol (#697)

* migrations (#697)

* Add translation to error meessage

* Make ASN readonly in table

* Add test now that ASN should not be able to update

* Set fac.rencode to '' for all entries and make it readonly in serializer

* Add unique constraints to network ixlan ip addresses

* Add migration to null out duplicate ipaddresses for deleted netixlans

* Add unique constraints to network ixlan ip addresses

* Add migration to null out duplicate ipaddresses for deleted netixlans

* remove old migrations (#697)

* fix netixlan ipaddr dedupe migration (#268)
add netixlan ipaddr unique constraint migration (#268)

* ixf_member_data migrations (#697)

* fix table name (#697)

* importer protocol (#697)

* fix netixlan ipaddr dedupe migration (#268)
add netixlan ipaddr unique constraint migration (#268)

* ixf proposed changes notifications (#697)

* Delete repeated query

* Add a test to show rencode is readonly

* Blank out rencode when mocking data

* Remove validator now that constraint exists

* Add back unique field validator w Check Deleted true

* conflict resolving (#697)

* UniqueFieldValidator raise error with code "unique" (#268)

* conflict resolution (#697)

* Add fixme comment to tests

* conflict resolution (#697)

* Remove now invalid undelete tests

* UniqueFieldValidator raise error with code "unique" (#268)

* delete admin tools for duplicate ip addresses

* Make migration to delete duplicateipnetworkixlan

* Add ixlan-ixpfx status matching validation, add corresponding test

* delete redundant checking in test

* resolve conflict ui (#697)

* fix migrations hierarchy

* squash migrations for ixf member data

* clean up preview and post-mortem tools

* remove non-sensical permission check when undeleting soft-deleted objects through unique integrity error handling

* only include the ix-f data url in notifications to admincom (#697)

* resolve on --skip-import (#697)

* ac conflict resolution (#697)

* Define more accurately the incompatible statuses for ixlan and ixpfx

* Add another status test

* Preventing disrupting changes (#697)

* fix tests (#697)

* Stop allow_ixp_update from being write only and add a global stat for automated networks

* Add tests for global stats that appear in footer

* Change how timezone is called with datetime, to get test_stats.py/test_generate_for_current_date to pass

* test for protected entities (#697)

* admincom conflict resolution refine readonly fields (#697)
network notifications only if the problem is actually actionable by the network (#697)

* ixp / ac notifcation when ix-f source cannot be parsed (#697)
fix issue with ixlan prefix protection (#697)

* migrations (#697)

* code documentation (#697)

* ux tweaks (#697)

* UX tweaks (#697)

* Fix typo

* fix netixlan returned in IXFMemberData.apply when adding a new one (#697)

* fix import log incosistencies (#697)

* Add IXFMemberData to test

* Update test data

* Add protocol tests

* Add tests for views

* always persist changes to remote data on set_conflict (#697)

* More tests

* always persist changes to remote data on set_conflict (#697)

* suggest-add test

* net_present_at_ix should check status (#697)

* Add more protocol tests

* Edit language of some tests

* django-peeringdb to 2.1.1
relock pipfile, pin django-ratelimit to <3 as it breaks stuff

* Add net_count_ixf field to ix object (#683)

* Add the IX-F Member Export URL to the ixlan API endpoint (#249)

* Lock some objects from being deleted by the owner (#696)

* regenerate api docs (#249)

* always persist changes to remote data on set_add and set_update (#697)

* IXFMemberData: always persist remote data changes during set_add and set_update, also allow for saving without touching the updated field

* always persist changes to remote data on set_add and set_update (#697)

* Fix suggest-add tests

* IXFMemberData: always persist remote data changes during set_add and set_update, also allow for saving without touching the updated field

* IXFMemberData: always persist remote data changes during set_add and set_update, also allow for saving without touching the updated field

* fix issue with deletion when ixfmemberdata for entry existed previously (#697)

* fix test_suggest_delete_local_ixf_no_flag (#697 tests)

* fix issue with deletion when ixfmemberdata for entry existed previously (#697)

* invalid ips get logged and notified to the ix via notify_error (#697)

* Fix more tests

* issue with previous_data when running without save (#697)
properly track speed errors (#697)

* reset errors on ixfmemberdata that go into pending_save (#697)

* add remote_data to admin view (#697)

* fix error reset inconsistency (#697)

* Refine invalid data tests

* remove debug output

* for notifications to ac include contact points for net and ix in the message (#697)

* settings to toggle ix-f tickets / emails (#697)

* allow turning off ix-f notifications for net and ix separately (#697)

* add jsonschema test

* Add idempotent tests to updater

* remove old ixf member tests

* Invalid data tests when ixp_updates are enabled

* fix speed error validation (#697)

* fix issue with rollback (#697)

* fix migration hierarchy

* fix ixfmemberdata _email

* django-peeringdb to 2.2 and relock

* add ixf rollback tests

* ixf email notifications off by default

* black formatted

* pyupgrade

Co-authored-by: egfrank <egfrank@20c.com>
Co-authored-by: Stefan Pratter <stefan@20c.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Development

Successfully merging a pull request may close this issue.

6 participants