Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[big!] Adds a completely re-made packages listing (explorer). #209

Merged
merged 8 commits into from Aug 30, 2019

Conversation

@samueldr
Copy link
Member

commented Apr 9, 2018

TLDR?

The code was previously present at : https://github.com/samueldr/nixos-packages-explorer

A demo (without the nixos website around): https://nix.samueldr.com/explorer/

EDIT: I have now realised that the demo does not show the odd/even lines as zebra-striped, like they do on the complete website. This is due to nixos-site.css not being present anymore in the stand-alone build.

Motivations

This is a re-made packages explorer which is re-made especially to add features, with the idea to make it easier to add additional new features.

The page is built in a way that ensures a link to the page carries the state over; ensuring that someone following a link has the same view you have (as long as the channel data is the same).

Furthermore, this can (I think) close those issues

  • #208 packages.html doesn't support "maintainers" object
  • #120 packages: include unfree and broken packages in the search index (partially, broken packages are not handled.)
  • #57 Add possibility to link to packages
  • #56 Add Package search to Nix homepage (though no page nor link has been added)
  • #53 Search NixOS packages doesn't show which channel is being searched
  • #49 "Search NixOS packages" doesn't show License
  • nixos#17126 Unfree software is unintuitively hidden for new users

Though some of those were already fixed anyway.

Additionally, a change fixes this issue (though it wasn't a goal)

  • #194 Provide archived "packages.json"
Unfree packages (#120 nixos#17126)

I do know that this is a thorny issue:

Sorry, but Nixpkgs is intended as a free software distribution, so it should not promote the availability of unfree packages. [...]
@edolstra

Though, I would like this PR not to be held up by this issue. If @edolstra so desires, I can remove the unfree checkbox until that discussion is resolved.

I, though, personally am in favor of allowing the user to choose whether they see unfree packages or not, and defaulting to them not showing up.

The way they are show (more grayed-out and in italics) can be changed, an additional descriptive text explaining what unfree is can also be added. Almost anything is possible here.

Broken packages (#120)

The feature is easy enough to add, if it is desired.

[...] It also shouldn't show broken packages unless they are conspicuously marked as broken.
@edolstra

Though, for this, I would like to have opinions and inputs about how (after this is merged) this could be implemented. Additionally (just an idea off the top of my head) could the brokenness include metadata with a textual description of how the package is broken?


Implementation notes

JavaScript

The application is built using preact, and as few other front-end dependencies as possible. The node_modules folder may be big, but most of this is for the build-time dependencies and development dependencies.

13,108 out of 15,629 "lines added" come from both the yarn.lock and yarn.nix files (5,381 and 7,727 respectively). This makes less than 2,521 lines for actual additions, of those some are boilerplate files and the rest is mostly ES6 files or Less files that are compiled and minified.

Makefile and build integration

I'm not that used to writing Makefiles, I'm happy to get schooled hear your inputs and fix the bad things with the Makefile.

Other than that, I'm pretty sure the build integration is alright. I made sure not to need nix-build inside the Makefile, as it reportedly breaks calling the Makefile from a nix expression. This is why there are two ways to build the packages explorer. The nixos-homepage way, with the Makefile, and the previous way I was building it, using nix-build and the release.nix file. The release.nix file can be dropped if wanted.


Repo notes

I have imported all the commits of the other repository. I understand that it is noisy as heck. It may be wise to squash all those before e749322.

It has been (partially) squashed through a rebase. It can be squashed even more if wanted.

@samueldr samueldr changed the title Adds a completely re-made packages listing (explorer). [big!] Adds a completely re-made packages listing (explorer). Apr 9, 2018
@davidak

This comment has been minimized.

Copy link
Member

commented Apr 9, 2018

Thanks for your work!

Here is a comparison:

screenshot_2018-04-09_21-15-39

Just some small notes:

  1. it would look cleaner without the lines (as before)
  2. i don't like the formatting of the longDescription. was better before
  3. on nixpkgs channel, install command is still NixOS, for example nix-env -iA nixos.ffmpeg (NixOS channel)
  4. on nixpkgs channel Platforms is empty
  5. for the channel selection i would like to have buttons or links (don't hide behind dropdown)

screenshot_2018-04-09_21-41-52

or

screenshot_2018-04-09_21-49-26

The Ubuntu package search for comparison:

screenshot_2018-04-09_21-55-28

Different header for Nix page:

screenshot_2018-04-09_21-53-32

Looks bad because i used code for bootstrap 4.

@samueldr

This comment has been minimized.

Copy link
Member Author

commented Apr 9, 2018

Thanks for the review!

1. Zebra-striping

  •  

To reduce chances of misunderstanding, I have re-imported the nixos-site.css file on my standalone deploy. This fixes the zebra-striping and some of the concerns. That was present only on the standalone deploy.

2. Long description

  •  

I have fixed the look for longDescription, good catch, I hadn't realised it wasn't a <pre> on the current site.

3. Install command

  •  

I would need to have the most likely name for the channel available in the channel data json file, and use it instead of heuristically try to guess it some other way.

This wouldn't be really hard to add, so let's assume that it can be added and will be.

4. Platforms being empty

  •  

Ah! 18.09 changes the structure of the fields for platforms, like 18.03 did for the maintainers object. I'll have to case it depending on the structure.


EDIT Oh my oh my, this may need some work on nixpkg's side. Here's what I get for platforms.unix:

[
  {
    "kernel": {
      "families": {
        "bsd": {
          "_type": "exec-format",
          "name": "bsd"
        }
      }
    }
  },
  {
    "kernel": {
      "families": {
        "darwin": {
          "_type": "exec-format",
          "name": "darwin"
        }
      }
    }
  },
  {
    "kernel": {
      "_type": "kernel",
      "execFormat": {
        "_type": "exec-format",
        "name": "elf"
      },
      "families": {},
      "name": "linux"
    }
  },
  {
    "kernel": {
      "_type": "kernel",
      "execFormat": {
        "_type": "exec-format",
        "name": "elf"
      },
      "families": {},
      "name": "solaris"
    }
  },
  {
    "kernel": {
      "_type": "kernel",
      "execFormat": {
        "_type": "exec-format",
        "name": "elf"
      },
      "families": {},
      "name": "hurd"
    }
  },
  {
    "abi": {
      "_type": "abi",
      "name": "cygnus"
    },
    "kernel": {
      "_type": "kernel",
      "execFormat": {
        "_type": "exec-format",
        "name": "pe"
      },
      "families": {},
      "name": "windows"
    }
  }
]

This is, uh, probably more verbose that what we want. We'll probably need some kind of "toString" that gives us something to print to the end-user.

5. Channel selection

  •  

Nothing is impossible, but the way I built it was to reduce the amount of information on-screen at once. Though it should be trivial to try out a different appearance (e.g. styled radio buttons; styled enough that the round button doesn't show up).

The goal is to make sure everything is accessible. E.g. the rows on the current listing aren't accessible using vimium (pressing F doesn't show a letter to access a row) or TAB-navigation. To smooth out the issue in the re-implementation I made sure to use an element that can be reached using keyboard-only navigation.

I'll (as the last thing to fix) look at alternatives to the select box.

@turion

This comment has been minimized.

Copy link

commented Apr 13, 2018

When I'm opening https://nix.samueldr.com/explorer/, it loads quite slowly. Is this because your personal server is slower than nixos.org, or is your javascript somehow less efficient?

@jtojnar jtojnar referenced this pull request Apr 16, 2018
3 of 8 tasks complete
@ryantm

This comment has been minimized.

Copy link
Member

commented Apr 16, 2018

Small grammar fix:
"This package is unfree. See chapter 6.2 of the manual for more informations."

"informations" -> "information"

@samueldr

This comment has been minimized.

Copy link
Member Author

commented Apr 16, 2018

@turion sorry for the delay, had a busy week-end. What do you mean "it loads slowly", is it stuck on the loading animation (three dots growing and reducing)? In that case, that means it's downloading.

My server does serve the file much slower than the nixos.org server serves the file. (2.18s vs. 0.511s!). Though, comparing the time after the channel data is downloaded, it doesn't look like it is slower to run.

Also note that there is an extra download that must be first done (+96ms here), which will always add the time for one server round-trip. That download is the list of channels that are available. This allows managing the available channels outside of the explorer's code.

@samueldr

This comment has been minimized.

Copy link
Member Author

commented Apr 16, 2018

Thanks @ryantm for the remark!

@samueldr samueldr force-pushed the samueldr:feature/packages-explorer branch from 206f4db to 5419ba3 Apr 16, 2018
@samueldr

This comment has been minimized.

Copy link
Member Author

commented Apr 16, 2018

20180416195438

This is also live on my demo page.

@matthewbauer

This comment has been minimized.

Copy link
Member

commented Apr 17, 2018

It looks like the old website would cache the .json file and yours is not? That could account for the performance differences.

Also, if you don't get much response on this, be sure to email the mailing list. I think this is in the right direction on fixing stuff. Some people might think it is too much new code to maintain and maybe it needs its own repo/external link.

@turion

This comment has been minimized.

Copy link

commented May 3, 2018

Yes, it seems the slow-down is just because the file is served slowly. Great otherwise!

fetch_channels() {
this.setState({loading: this.state.loading + 1});

return fetch("/nixpkgs/packages-channels.json", {mode: "cors"})

This comment has been minimized.

Copy link
@jcrben

jcrben May 6, 2018

this file can change over time right? seems like you'd want hash here to cache bust this file when the channel has been updated, otherwise I think the browser will use the cached version. same applies to the below json file. https://survivejs.com/webpack/optimizing/adding-hashes-to-filenames/

but looks the current site doesn't version the files either - I might be missing something.

@jcrben

This comment has been minimized.

Copy link

commented May 6, 2018

overall seems like a solid improvement - the current version is pretty barebones and also loads slow

Stack choice is familiar and I at least like it. in case you want to write some tests I like the Jest framework - jsdom gives you some great power to build non-fragile UI tests. also recommend prettier

long-term, not sure that loading the entire packages file is the best approach, especially when (1) there's pagination to show only 15 results and clicking on a separate distribution fetches another big file and (2) the file could grow far larger as time passes. ideally seems like you could efficiently filter the results and render the page on the server and then fetch the rest as needed when the client clicks thru pages. but I don't want to derail your work and not sure what the view around here is on a server

the slow load for me:
image

1200 kilobytes / 21 seconds = 57 kilobytes/s
seems like your server really doesn't like sending stuff to my client - but on refreshes seems like got to a more reasonable speed 😄

don't have a ton of capacity to help but feel free to hit me up publicly or privately if you want to chat frontend

@Infinisil

This comment has been minimized.

Copy link
Member

commented Aug 4, 2018

The current demo makes a new history entry on every button press, which means I had to go "back" in the browser like 20 times to actually get to the last website after playing around for a bit.

I don't know much about javascript, but I think such instant-transition-inducing button presses shouldn't cause new history entries. In the users (my) eyes, it is still the same webpage, so it shuoldn't do that. I occasionally happen to find other websites online that do this, but was never fond of that.

@samueldr

This comment has been minimized.

Copy link
Member Author

commented Aug 4, 2018

(Related IRC log for the record)

Yeah, though your argument is artificially limited to something you feel.

Let's dwell only on the pager previous/next page for now.

Technically speaking, when you click on the "next" button on the issues page, do you expect it to push an navigation event to your history or not? You probably do. Well, the load, while not entirely instantaneous, isn't a full page load; JavaScript did things and only made partial changes to the page.

In this re-implementation of the searcher, clicking on the "next" button, you wouldn't expect to have a navigation event pushed on the stack. In this particular case, the load is a bit more instantaneous, but the same mechanism are in use; JavaScript does things and makes a partial change to the page.

AFAICT (and I'm not trying to be contrarian here!) the only difference in the pager is that one is wicked fast while the other is slower. The github page load is slower since it fetches the data in the background (still via JavaScript). The package searcher is faster since all of the data is pre-loaded due to how we lazily load all the data upfront.

Here's a hill I'm prepared to get bruised on: the behaviour of a webpage shouldn't change because of the way the back-end has been architected. Following a link should add a navigation event to the history, and change the URL accordingly; the changed URL should load the site to work the same way if accessed directly.


Now: here's to the one issue I concede is a hard one to solve right:

When should a navigation event be made on the history stack for a live-changing result.

The best solution I found from experience is to pop on blur1 (and on occasion, blur on enter press). It feels fine to me to expect that on blur, the user has "committed" the changes and is ready to peruse the results.

Not touching the history in that case will cause weird issues considering other actions touch the history. You would move back and forth into the history and be missing the first page of the results, it would be:

  • foo 2
  • foo 3
  • foo 4
  • bar 2
  • bar 3

Instead of:

  • foo 1
  • foo 2
  • foo 3
  • foo 4
  • bar 1
  • bar 2
  • bar 3

The "dumb" solution to this problem is to reduce the apparent functionality and force committing the change. Instead of live searching and updating the whole page, showing suggestions in another control/component, allowing completion, and then asking for an action (pressing enter, clicking a Search button) to commit the new search; thus pushing the new navigation event to "page 1 of new search".


1: blur, when the focus exits the field. Either by tab navigation, clicking elsewhere, etc.

@davidak

This comment has been minimized.

Copy link
Member

commented Jan 30, 2019

@samueldr what's the status here?

I would love to have it! (especially archived packages.json for repology)

@samueldr

This comment has been minimized.

Copy link
Member Author

commented Jan 30, 2019

Still waiting for comments from those in charge of the site. I'm still unclear as to whether the feature is desired, whether the implementation is fine, what needs to be done to upstream this. I pretty much haven't touched it at all since not knowing could mean efforts go in the wrong way with regards to what's desired.

I'm willing to polish this up any way desired, even re-do parts of it if needed, but I need some direction to do this.

@ryantm

This comment has been minimized.

Copy link
Member

commented Jan 30, 2019

cc @edolstra @zimbatm as people in charge of this site (I'm not sure if there are others).

@samueldr

This comment has been minimized.

Copy link
Member Author

commented Jan 30, 2019

To add to the previous comment: I'm glad to work with you for any issues, perceived or real about this PR. After all, this adds a big chunk of machinery in the website build, and is probably not optimally done.

@Ekleog

This comment has been minimized.

Copy link
Member

commented Mar 12, 2019

@edolstra @zimbatm would it be possible to move this forward? @samueldr is ready to continue work on this PR, but would need some direction as to what's OK and what's not in the current state of this PR, if I interpret the last comment correctly.

@zimbatm

This comment has been minimized.

Copy link
Member

commented Mar 14, 2019

This is really good work and I would be happy to merge the PR as is (once the merge conflicts are fixed).

In the long term, I would like to see the NixOS options, nixpkgs packages.json and manuals be published by the channel release script instead and have the nixos.org website be essentially a static page that loads the JSON using CORS. It would be nice if that was also tackled here but I understand that it would expand the scope of the PR.

@davidak

This comment has been minimized.

Copy link
Member

commented Jul 18, 2019

@samueldr can you resolve the merge conflict? 🚀

@flokli flokli referenced this pull request Jul 20, 2019
6 of 10 tasks complete
@zimbatm

This comment has been minimized.

Copy link
Member

commented Aug 19, 2019

I tested locally and it looks great!

@samueldr what does the (NixOS Channel) mean next to the install command? Eg:

image

Copy link
Member

left a comment

thanks, this is great!

@samueldr

This comment has been minimized.

Copy link
Member Author

commented Aug 19, 2019

@samueldr what does the (NixOS Channel) mean next to the install command?

This comes from the existing packages page. I'm not entirely sure what it should mean. We may as well drop that.

@zimbatm

This comment has been minimized.

Copy link
Member

commented Aug 19, 2019

If you feel like it. Things are already much better like that.

@zimbatm

This comment has been minimized.

Copy link
Member

commented Aug 19, 2019

I also have my eyes on the maintainers field. It would be nice to get the github handle or a link to the maintainer meta attributes instead of just the name and email.

@samueldr

This comment has been minimized.

Copy link
Member Author

commented Aug 19, 2019

Improvements to build upon :)

(Yes, good ideas, and things that should be easier to implement in the end.)

@grahamc

This comment has been minimized.

Copy link
Member

commented Aug 20, 2019

When turning on unfree packages, I'm finding the contrast of an unfree package to be hard to read on odd zebra striped package details. The text is #666666, the background is F5F5F5. It does pass minimum contrast accessibility requirements, though, so I won't block on it.

</table>
<div class="result--permalink">
<Link merge={true} state={{attr}}>
Link to <tt>{attr}</tt>

This comment has been minimized.

Copy link
@grahamc

grahamc Aug 20, 2019

Member

I really like this, but was confused (honestly), and until I inspected the link, I wasn't sure what it did. I wondered:

  • is it taking me to the package's homepage?
  • is it a link to the hydra job?
  • is it taking me to the expression?
  • is it a permalink to this entry?
  • is it going to copy the permalink to this entry automatically?

I think the button is probably a bit big for what it actually is (a permalink to the package), and maybe the text should just say "permalink". What do you think?

side note, I imagine it is tricky, but having it auto-copy the URL would be fancypants.

This comment has been minimized.

Copy link
@samueldr

samueldr Aug 20, 2019

Author Member

As we make sure not to thrash the user's history, I think it is better not to thrash the user's clipboard unexpectedly. Users can right click and copy link location, or follow the link and copy what is in their location bar. Copying a permalink to the clipboard automatically started as a poor workaround for web application unable to handle proper back/forward navigation events.

@samueldr

This comment has been minimized.

Copy link
Member Author

commented Aug 20, 2019

About unfree software

The project's view about unfree software is known. We shouldn't promote unfree software.

This rewrite of the packages-explorer was written in a way that is easy to extend, and any part can easily be removed or tweaked as desired, including the filter for unfree software (which defaults to hide them). The cost of adding such a filter is much reduced compared to the previous codebase.

Still, my personal opinion is that it is a usability issue not to list unfree software in some way. Though, I do agree that we shouldn't promote unfree software, which is why they are gated under a toggle, and phrasing is added to every unfree software entry. The tone of that note was initially more neutral-to-negative by design.

Seeing that the foundation depends on unfree software, it seems weird to me to completely hide the presence of unfree software from the packages set, acting as if it didn't exist.


As a closing note, more details about my view. A part of Freedom, is the Freedom of choice, including the choice of using unfree software. We should better educate the user so they make an informed decision about their choice of using unfree software, and we can make it obvious what the position of the project is. Hiding the fact that the project has unfree software, while still packaging it is probably unwise, and does reduce usability for the users that have decided that the drawbacks of unfree software is worth it for their needs. Some users probably have skipped this distribution, thinking that they couldn't use some unfree software, possibly a hardware driver.

@samueldr samueldr force-pushed the samueldr:feature/packages-explorer branch from 0c164f4 to f2659aa Aug 29, 2019
@samueldr

This comment has been minimized.

Copy link
Member Author

commented Aug 29, 2019

I have just now cut off the unfree part, part of #300 that is meant to apply on master once this is merged.

@samueldr samueldr force-pushed the samueldr:feature/packages-explorer branch from f2659aa to ddc4c6d Aug 29, 2019
EXPLORER_JS = $(shell find packages-explorer/ -not -path 'packages-explorer/node_modules/*')

nixos/packages-explorer.js: $(EXPLORER_JS)
(cd packages-explorer ; nix-build -I nixpkgs=$(CHANNEL_NIXOS_STABLE))

This comment has been minimized.

Copy link
@edolstra

edolstra Aug 30, 2019

Member

Hm, I'd prefer not to call nix-build from the Makefile because that makes it harder to have a reproducible build of the homepage.

For background, the flake branch of nixos-homepage adds a Nix build that calls the Makefile (see https://github.com/NixOS/nixos-homepage/blob/flake/flake.nix), so if the Makefile calls Nix, that won't work anymore.

This comment has been minimized.

Copy link
@samueldr

samueldr Aug 30, 2019

Author Member

I do agree. I initially asked for help in figuring out a nix-build-free solution, but was told to continue with it. Though I think it should be possible to do the same thing that is done with the documentation updates I've been working on with @grahamc. Basically, have a full-fledged derivation that is symlinked in place during the nix build, and nix-built otherwise. It should be fixable relatively easily.

This comment has been minimized.

Copy link
@edolstra

edolstra Aug 30, 2019

Member

This was pretty easy to fix, flake.nix now imports/builds the package-explorer Nix expression directly and passes the build result to the Makefile(

packagesExplorer = import ./packages-explorer nixpkgsStable;
).

@edolstra edolstra merged commit ddc4c6d into NixOS:master Aug 30, 2019
@samueldr samueldr deleted the samueldr:feature/packages-explorer branch Aug 30, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.