New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Federation improvements #5209

Merged
merged 9 commits into from Sep 27, 2014

Conversation

Projects
None yet
3 participants
@jhass
Copy link
Member

jhass commented Sep 6, 2014

See commits

add pry and pry-debundle to the Gemfile
If you now install pry-byebug and pry-stackexplorer,
you'll get a nice debugger

@jhass jhass changed the title Federation improvements [WIP] Federation improvements Sep 6, 2014

@jhass jhass force-pushed the jhass:federation_improvements branch 2 times, most recently from d02e856 to 3a355ad Sep 6, 2014

@jhass jhass added this to the next-major milestone Sep 6, 2014

jhass added some commits Sep 6, 2014

Switch Faraday adapter to typhoeus
It uses curl which has less problems connecting
to a missconfigured IPv6 host (falls back to v4)
Try fetching missing parent of relayables
* Extract post fetching logic from Reshare into
  its own module
* raise proper error message when fetching fails
* raise proper error message when parent is still missing

We can't skip fetch failures or missing parents and
still need to retry them in case we're sent the parent
later on

@jhass jhass force-pushed the jhass:federation_improvements branch from 0cf5b3f to 619bc3f Sep 7, 2014

@Raven24

This comment has been minimized.

Copy link

Raven24 commented on lib/diaspora/fetcher/single.rb in 777e312 Sep 7, 2014

don't you have to specify a symbol to module_function?

This comment has been minimized.

Copy link

Raven24 replied Sep 7, 2014

nvm, just wasn't sure what it does ;)

@Raven24

This comment has been minimized.

Copy link
Member

Raven24 commented Sep 7, 2014

so I feel this should improve at least the circulation of public content significantly.
any relayables parent will now be fetched if it's not there already (and it is public)

@jhass

This comment has been minimized.

Copy link
Member

jhass commented Sep 7, 2014

Yep, that's the idea. My dead queue went to to 60 from 280, gonna see how the failures develop. There's a race condition where the same post is fetched twice in parallel or something so sometimes the uniqueness validation fails while inserting, though the retries take care of that.

Gonna keep this running for a couple of days on my pod to see whether I find any negative side effects.

@Flaburgan

This comment has been minimized.

Copy link
Member

Flaburgan commented Sep 10, 2014

Wow, thank you for that patch! Everything which makes the retries go down is really welcome. I guess this has to be ported to the gem too.

@jhass jhass referenced this pull request Sep 13, 2014

Closed

[WIP] Global hydra #5217

@jhass

This comment has been minimized.

Copy link
Member

jhass commented Sep 13, 2014

Didn't notice any obvious regressions so far, so should be safe for larger testing.

Average failure rates seem to be down quite a bit (marked datapoint is first day fully on this branch):

@jhass jhass changed the title [WIP] Federation improvements Federation improvements Sep 18, 2014

@Flaburgan

This comment has been minimized.

Copy link
Member

Flaburgan commented Sep 18, 2014

This is currently tested in production on diaspora-fr. Let's wait a week and see how things are going.

@Flaburgan

This comment has been minimized.

Copy link
Member

Flaburgan commented Sep 26, 2014

sidekiq

Okay we can clearly say that this patch is a success, I don't even need to point on the graph where the patch was applied because the drop is really visible. Thank you Jonne for this work!!

@Raven24

This comment has been minimized.

Copy link
Member

Raven24 commented Sep 26, 2014

ok, so I think we have a really helpful improvement here.
@jhass do you wanna clean up the commits a bit or should we merge it as-is to have a descriptive history?

@jhass

This comment has been minimized.

Copy link
Member

jhass commented Sep 26, 2014

No fixup commits of previous commits in there, I'd rather keep that bisect'able ;)

Raven24 added a commit that referenced this pull request Sep 27, 2014

@Raven24 Raven24 merged commit c7b4b77 into diaspora:develop Sep 27, 2014

1 check passed

continuous-integration/travis-ci The Travis CI build passed
Details
@Raven24

This comment has been minimized.

Copy link
Member

Raven24 commented Sep 27, 2014

alright, merged.
thanks!

@jhass jhass deleted the jhass:federation_improvements branch May 5, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment