New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Check completeness of new site #150

Closed
wolfgangmm opened this Issue Mar 15, 2016 · 17 comments

Comments

Projects
None yet
4 participants
@wolfgangmm
Contributor

wolfgangmm commented Mar 15, 2016

We need to make sure the new site has all the pages which were available on the old site. Can we create a crawler to collect all links on the new site, then run them against the new site? I did something like this in nodejs a while ago, but lost the code.

@line-o

This comment has been minimized.

Show comment
Hide comment
@line-o

line-o Mar 23, 2016

Contributor

We used to to that with wget, long ago.

Contributor

line-o commented Mar 23, 2016

We used to to that with wget, long ago.

@line-o line-o self-assigned this Mar 23, 2016

@line-o

This comment has been minimized.

Show comment
Hide comment
@line-o
Contributor

line-o commented Mar 23, 2016

@line-o

This comment has been minimized.

Show comment
Hide comment
@line-o

line-o Mar 30, 2016

Contributor

Finally, some results:
/open is missing
/gallery and /gallery/south-east-asia-conference
/countries/issues and 3 subpages
around 2000 peoples pages still missing
Missing pages are stored in
not-found.txt

Contributor

line-o commented Mar 30, 2016

Finally, some results:
/open is missing
/gallery and /gallery/south-east-asia-conference
/countries/issues and 3 subpages
around 2000 peoples pages still missing
Missing pages are stored in
not-found.txt

@joewiz

This comment has been minimized.

Show comment
Hide comment
@joewiz

joewiz Mar 30, 2016

Member

To confirm, these are the results for the non-/historicaldocuments/frus* pages, correct?

Member

joewiz commented Mar 30, 2016

To confirm, these are the results for the non-/historicaldocuments/frus* pages, correct?

@plutonik-a

This comment has been minimized.

Show comment
Hide comment
@plutonik-a

plutonik-a Mar 30, 2016

Contributor

Page /open is definitely not missing, at least I worked on it a minute ago on branch fix-grid-layout and it's linked from the main menu under "More Resources".

http://localhost:8080/exist/apps/hsg-shell/open
https://history.state.gov/beta/open

Contributor

plutonik-a commented Mar 30, 2016

Page /open is definitely not missing, at least I worked on it a minute ago on branch fix-grid-layout and it's linked from the main menu under "More Resources".

http://localhost:8080/exist/apps/hsg-shell/open
https://history.state.gov/beta/open

@joewiz

This comment has been minimized.

Show comment
Hide comment
@joewiz

joewiz Mar 30, 2016

Member

The /gallery result is expected, and /countries/issues is quite possible too. But, like @plutonik-a's note about /open, I wonder about the people results. For example, one of the not-found.txt entries is found on the beta site: https://history.state.gov/beta/departmenthistory/people/ackerman-ralph-h (and I confirmed the response code is 200).

@line-o Is it possible your hsg-shell (or hsg-project) is out of date?

Member

joewiz commented Mar 30, 2016

The /gallery result is expected, and /countries/issues is quite possible too. But, like @plutonik-a's note about /open, I wonder about the people results. For example, one of the not-found.txt entries is found on the beta site: https://history.state.gov/beta/departmenthistory/people/ackerman-ralph-h (and I confirmed the response code is 200).

@line-o Is it possible your hsg-shell (or hsg-project) is out of date?

@line-o

This comment has been minimized.

Show comment
Hide comment
@line-o

line-o Mar 30, 2016

Contributor

I rebuild data & database quite frequently. Last time was yesterday. But you are both right they are on the beta site. Very strange.

Contributor

line-o commented Mar 30, 2016

I rebuild data & database quite frequently. Last time was yesterday. But you are both right they are on the beta site. Very strange.

@line-o

This comment has been minimized.

Show comment
Hide comment
@line-o

line-o Mar 30, 2016

Contributor

That is what happens on my machine:
screen shot 2016-03-30 at 16 35 24

Contributor

line-o commented Mar 30, 2016

That is what happens on my machine:
screen shot 2016-03-30 at 16 35 24

@line-o

This comment has been minimized.

Show comment
Hide comment
@line-o

line-o Mar 30, 2016

Contributor

I can now confirm my local version was outdated. Will re-run the tests!

Contributor

line-o commented Mar 30, 2016

I can now confirm my local version was outdated. Will re-run the tests!

@line-o

This comment has been minimized.

Show comment
Hide comment
@line-o

line-o Mar 30, 2016

Contributor

Here is the very short new list:

  • /countries/issues
  • /countries/issues/china-us-relations
  • /countries/issues/german-unification
  • /countries/issues/italian-unification
  • /gallery
  • /gallery/southeast-asia-conference
  • /gallery/southeast-asia-conference?group=iconic
  • /gallery/southeast-asia-conference?group=media
  • /gallery/southeast-asia-conference?group=notable-people
  • /gallery/southeast-asia-conference?group=on-the-ground
  • /gallery/southeast-asia-conference?group=peace-negotiations
  • /gallery/southeast-asia-conference?group=protest
  • /historicaldocuments/guide-to-sources-on-vietnam-1969-1975
Contributor

line-o commented Mar 30, 2016

Here is the very short new list:

  • /countries/issues
  • /countries/issues/china-us-relations
  • /countries/issues/german-unification
  • /countries/issues/italian-unification
  • /gallery
  • /gallery/southeast-asia-conference
  • /gallery/southeast-asia-conference?group=iconic
  • /gallery/southeast-asia-conference?group=media
  • /gallery/southeast-asia-conference?group=notable-people
  • /gallery/southeast-asia-conference?group=on-the-ground
  • /gallery/southeast-asia-conference?group=peace-negotiations
  • /gallery/southeast-asia-conference?group=protest
  • /historicaldocuments/guide-to-sources-on-vietnam-1969-1975
@joewiz

This comment has been minimized.

Show comment
Hide comment
@joewiz

joewiz Apr 6, 2016

Member

@plutonik-a Just curious, what is the relationship with #122?

Member

joewiz commented Apr 6, 2016

@plutonik-a Just curious, what is the relationship with #122?

@plutonik-a

This comment has been minimized.

Show comment
Hide comment
@plutonik-a

plutonik-a Apr 6, 2016

Contributor

That was most definitely an error, must have mixed up the urls of the issues.

Contributor

plutonik-a commented Apr 6, 2016

That was most definitely an error, must have mixed up the urls of the issues.

@joewiz

This comment has been minimized.

Show comment
Hide comment
@joewiz

joewiz Apr 6, 2016

Member

Okay, no worries!

Member

joewiz commented Apr 6, 2016

Okay, no worries!

@joewiz joewiz modified the milestones: 1.0 - Launch, 1.0 - Launch beta, 1.1 - Complete beta Apr 22, 2016

@joewiz joewiz added high priority and removed high priority labels May 24, 2016

@line-o

This comment has been minimized.

Show comment
Hide comment
@line-o

line-o May 26, 2016

Contributor

Should we do a re-run and expect /gallery and /countries/issues now to be there as well?

Contributor

line-o commented May 26, 2016

Should we do a re-run and expect /gallery and /countries/issues now to be there as well?

@line-o

This comment has been minimized.

Show comment
Hide comment
@line-o

line-o May 26, 2016

Contributor

Otherwise we can close this issue and create a new one for those.

Contributor

line-o commented May 26, 2016

Otherwise we can close this issue and create a new one for those.

@joewiz

This comment has been minimized.

Show comment
Hide comment
@joewiz

joewiz May 26, 2016

Member

They're covered with existing issues: #97 for /gallery and #184 for /countries/issues. Once those are complete, a re-run would be a great step to take before removing the "beta" label from the new site.

Member

joewiz commented May 26, 2016

They're covered with existing issues: #97 for /gallery and #184 for /countries/issues. Once those are complete, a re-run would be a great step to take before removing the "beta" label from the new site.

@joewiz

This comment has been minimized.

Show comment
Hide comment
@joewiz

joewiz Mar 24, 2017

Member

Closing this issue, since a full re-check of all site URLs is not necessary. The list above sufficed, and I've added those URLs to the relevant issues.

Member

joewiz commented Mar 24, 2017

Closing this issue, since a full re-check of all site URLs is not necessary. The list above sufficed, and I've added those URLs to the relevant issues.

@joewiz joewiz closed this Mar 24, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment