Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RRDP files are deleted immediately after use #81

Open
ydahhrk opened this issue Mar 11, 2022 · 1 comment
Open

RRDP files are deleted immediately after use #81

ydahhrk opened this issue Mar 11, 2022 · 1 comment

Comments

@ydahhrk
Copy link
Member

ydahhrk commented Mar 11, 2022

(RRDP files = Notifications, Snapshots and Deltas.)

It seems to have been implemented this way to preserve disk space, but has two drawbacks I find impractical:

  1. It complicates debugging; it means if there's a problem, I just lost the file that triggered it.
  2. It's (presumably) the reason why session IDs and serials need to be cached in RAM. (Prevent redownload of all RRDP on startup #79)

Branch rrdp-refactor has a WIP of this bug.
ydahhrk added a commit that referenced this issue Sep 9, 2023
@ydahhrk
Ongoing rewrite of the local cache code
c717043 
It's a bit smarter now. Addresses a bunch of issues at once, though it
still needs several tweaks and testing:

- #78: Provide a dedicated namespace for each RRDP notification, to
  prevent malicious RPPs from overriding files from other RPPs.
- #79: RRDP session and serial are no longer cached in RAM; they're
  extracted from cached notification files as they are needed.
  This prevents all RRDP from being considered outdated during startup.
- #80: rsync-strategy has been removed.
- #81: The cache now retains RRDP files.

The refactor has been more intrusive than intended. I've been retouching
the core loop and rrdp/https code, which has yielded the following
further disinfections:

- #77: Refactor the HTTP code so 304 is handled as success, despite no
  file modifications having been made.
- It seems the old code was refusing to download RPPs via RRDP when said
  RPP wasn't also (unrelatedly) served via rsync. This seemed to stem
  from an old RFC misunderstanding from the previous developer.
- I've deprecated `rsync.priority` and `rrdp.priority`, mostly just to
  simplify the code. I haven't seen anyone using these config fields,
  and I think SIAs and/or randomness should be the ones to decide which
  protocol is preferred for a given RPP, not Fort's admin.
- However, I have also decided to deprecate `shuffle_tal_uris`, because
  I also suspect it's completely unused, and would like to hear some
  complaints otherwise.
- Deprecated `rsync.arguments-flat`, because non-recursive rsyncs are
  not needed anymore.
- Since RRDP files are no longer deleted immediately after use, the
  `DEBUG_RRDP` compilation has lost its purpose, so I deleted it.
- The code was using `HASH_ADD_STR` on strings contained outside of the
  node structure. This is illegal according to uthash's documentation,
  and might have induced some crashes in the past.
@ydahhrk ydahhrk added this to the v1.6.0 milestone Oct 6, 2023
@ydahhrk
Copy link
Member Author

ydahhrk commented Dec 1, 2023

In the end, I decided to address this differently.

Keeping RRDP files in the cache (even temporarily) is not the wisest idea, obviously. Their disk usage is not insignificant.

I started designing metadata files that would provide the benefits of both worlds, but this grew outside of the scope of 1.6.0.

So, though there's already a startup in 1.6.0, this will remain open for a while longer.

@ydahhrk ydahhrk removed this from the v1.6.0 milestone Dec 1, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ydahhrk